tj0

200 Sucess

2023-02-09T12:06:29-08:00

200 Success

Mapping repository dependencies with Github CLI and Python

I transitioned into my new role as an application security engineer, and immediately started diving in on a Shift Left effort. My goal for the first quarter is to get static analysis running on every developer’s local system using semgrep. I had hit the ground running, was making progress, and as per usual in infosec we had an urgent request come in for a potential client.

The request was simple enough, something along the lines of “Legal needs a comprehensive list of all the open source dependencies our software uses. We’re trying to get the contract out within 30 days.” Cool, cool. So basically a promise was made to the potential client and we have to deliver this artifact like, last week.

I figured a request like this is common enough, and we even have a tool called Cider which provided supply chain data. So, trying to be efficient I thought, well let’s just export the list of all the things, narrow the list down to production only repos, then de-duplicate. Done! But, as any senior engineer knows if it’s that easy, something isn’t right. Well, in this case when trying to export a CSV with 36k records, the request times out on Cider.

Ok, so that won’t work. I’ll just use the Github API, and I’ve been wanting to expand my golang chops. This should be a great opportunity for that!

Well, golang is kind of an asshole and picking up some syntactical details I’d glossed over in the Golang for dummies tutorials I’ve read came back to bite me. Finally I got something going but there was a problem — my personal access token was only fetching my personal repos on Github, not the repos associated with my organization. Off to another adventure into the depths of Github documentation to figure out something else I didn’t know I didn’t know, which was cool because now our organization has PAT access hardened (another quick win).

TL;DR: I was finally able to fetch all the repos within my organization using https://pkg.go.dev/github.com/google/go-github/v50/github.

BUT another problem — there’s no f’ing way to fetch a repository’s dependency map via Github’s RESTful API :facepalm:. To do that, you have to use the GraphQL API, which was yet ANOTHER language I had to pick up. Truth be told, I gave up on this effort after about three hours of head bashing and much colorful language.

Then something amazing happened… a fellow engineer cough thanks @spaceB0xx cough sent me this link https://github.com/andyfeller/gh-dependency-report and said “maybe this will help.” A quick review of the code sparked my interest and I went to bed hopeful. The next morning I got up, logged on, and ran:

gh extension install andyfeller/gh-dependency-report
gh dependency-report Organization backend-service

A REPORT WAS GENERATED! I vengefully deleted the three directories containing various attempts for doing what this extension did in about 30 seconds, and proceeded to run:

gh dependency-report Organization
^C

I Ctrl-Ced as soon as the massive list of 600+ repositories appeared as we have a ton of dead repos, internal tooling repos, playgrounds, etc. I only needed ~170 of these repositories in the resulting report so after cleaning up that list I was able to run the extension with the admittedly still massive list of repos and go for a walk. When I returned, we finally had a 25MB file containing a list of all production-impacting dependencies and I was on to the easy part: de-duplicate the list, break the list down into separate files containing language specific dependencies, remove internal dependencies, and produce one last file listing the unique licenses on each of those dependencies.

I won’t cover what this code does line for line, but after a couple hours of hacking this thing together we have our deliverable in hand, and tooling to make sure when the next request of this kind comes in we aren’t scrambling like we were this time round.

import csv
import pandas as pd

FILE_NAMES = {
    'PIP': 'python_dependencies',
    'NPM': 'javascript_dependencies',
    'COMPOSER': 'php_dependencies',
    'NUGET': 'dotnet_dependencies',
    'ACTIONS': 'github_dependencies',
    'RUBYGEMS': 'ruby_dependencies'
}

with open('full_dependency_list-prod-repos.csv') as file_in:
    print('[!] Processing report....')
    csv_in = csv.DictReader(file_in)
    keep_fields = ['Dependency', 'Version', 'License Type', 'License URL']
    outputs = {}

    licenses = set()

    for row in csv_in:
        pkg_manager = row['Package Manager']
        # open new file and write the header
        if pkg_manager not in outputs:
            file_name = pkg_manager
            if pkg_manager in FILE_NAMES:
                file_name = FILE_NAMES.get(pkg_manager)

            print(f'[!] Creating {file_name}.csv....')
            file_out = open(f'{file_name}.csv', 'w', newline='')
            dw = csv.DictWriter(file_out, fieldnames=keep_fields)
            dw.writeheader()
            outputs[pkg_manager] = file_out, dw

        # fetch only the fields we want
        fields_to_write = {
            'Dependency': row.get('Dependency'),
            'Version': row.get('Requirements'),
            'License Type': row.get('License'),
            'License URL': row.get('License Url')
        }

        # write the row
        outputs[pkg_manager][1].writerow(fields_to_write)
        licenses.add(f'{row.get("License")} {row.get("License Url")}')

    # close all files
    for file_out, _ in outputs.values():
        print(f'[!] Closing {file_out.name}...')
        file_out.close()

    # generate licenses file
    license_out = open('unique_licenses.csv', 'w', newline='')
    license_header = ['License Type', 'License URL']
    dw = csv.DictWriter(license_out, fieldnames=license_header)
    dw.writeheader()

    for lic in licenses:
        temp = lic.split(' ')
        fields_to_write = {
            'License Type': temp[0],
            'License URL': temp[1]
        }
        dw.writerow(fields_to_write)

    license_out.close()

for f in FILE_NAMES:
    file_name = FILE_NAMES.get(f)
    print(f'[!] De-duplicating {file_name}.csv....')
    df = pd.read_csv(f'{file_name}.csv')
    df.drop_duplicates(inplace=True)
    df.to_csv(f'{file_name}.csv', index=False)

print('[!] Done!')

Hopefully this helps someone else out there faces with a similar problem. Thanks for reading.

404 Not Found

2023-01-04T15:20:29-08:00

404 Not Found

It’s been a while…

This place is dusty. I’ve been MIA for a while, at least on the blogging front. Despite that, I remembered, “oh shit, I have analytics for my blog, I wonder how it’s been doing.” 2022 was a wild year, to say the least. In January traffic peaked out at just shy of 1400 visitors on the biggest day, and in July… I have no idea what was going on but there were several 5,000+ visitor days. Anyways, that made me consider whoever is actually reading this content might be curious where the hell I’ve been.

I’ve been everywhere, man.

July of 2021, I accepted a new position as a senior software engineer on a platform team at a pretty cool company. The team I was on was composed of some of the best engineers I’ve ever had the privilege of working with, the company is 100% remote first, and the best part is they saved me from suffering in PHP land any further. I FINALLY ESCAPED THE TRAP! Python and Node are the flavors of choice here, and yes I’m speaking in present tense because I’m still with the same company.

All hasn’t been great, though. Within two months of my start date my EM and principal engineer left. Two months later, two other senior engineers left. And finally, in December the last remaining original team member departed. Talk about a soft landing — I show up and everyone leaves the room. Awkward…

In the midst of this chaos I did see an opportunity though. For the previous couple years I had entertained the idea of transitioning into people leadership. The reasons were varied, and somewhat driven by what a typical technical tract looks like — you get to senior, sit there for half a decade, then maybe make staff, then after another decade maybe make principal. All the while salaries stagnate at a certain point, and to me, you end up picking up a ton of people leadership responsibilities while still juggling the expectations of being highly productive. Why would you want to be accountable for people reporting to you while also juggling exponentially more technical problem scopes? Not for me, or at least I thought that then.

Instead, as my team hired on two new engineers and brought two others from another team into the fold, I stepped up as the team lead. Given the team’s EM was technically the entire platform organization’s director, I was picking up a ton of EM tasks and progressively being pulled out of heads down coding time. After months of doing that, my team and I pushed to make me the official EM. #careermilestoneachieved!

For those of you considering this transition, take heed to this next piece of advice.

Transitioning to leadership is way harder than you comprehend right now, and it is not for the faint of heart. Expect your working hours to increase by at least 50% initially, and to make more mistakes than you’ve made since you were a junior engineer. Don’t expect to just pick up the title and run. I know you won’t listen, but I feel compelled to tell others considering this transition as the time since my promotion to EM has had a lot of negative impacts on me personally despite providing exponential opportunities for growth that I didn’t know existed. I’ll write about all that at a later time, though.

This brings me to now.

Given I’ve figured out being in people leadership isn’t what I want to do, at least not right now, I interviewed for an internal senior appsec engineer role and received the offer. I’m finally getting back to what I was doing before I left the last company, and getting to repeat almost exactly what I did there, just here. The fact that this company provides so much flexibility, and opportunity, for their people to do the things they’re passionate about while enabling employee’s personal growth is by far my favorite trait about this place. I’ll be stepping down as the EM for my team (much to their disappointment) at the end of January, and I cannot express how excited I am to have some semblance of work life balance back. To be able to put my work down at the end of the day. To be fully present with my kids and partner again. Plus, I’ll be getting to engage in offensive security shenanigans while helping engineers throughout the entire engineering organization level up.

That’s it. That’s the TL;DR: of what I’ve been up to. I’ll be dropping some additional entries alongside this one that I’ve had in draft state and just haven’t touched in some time. I know this one is personal, and atypical of most of my other content, but at this point I’m going to be pivoting this blog to be a public journal. Maybe I’ll be more consistent. Unlikely, but maybe.

Take care.

500 Internal Server Error

2021-01-15T19:13:00-08:00

500 Internal Server Error

A malformed request. A stray character. An array where an object should be. The wrong HTTP request type. We’ve all experienced 500 errors when integrating with a new API, or an old API. We’ve seen 500’s when doing our own development; dived into logs, added print_r and echo at each step. Pulled out a debugger, even. gdb or xdebug. In pursuit of finding the how to the why, trying to understand.

But what happens when our brain returns a 500? How do you deal with that? What do you call it? We’re not servers. Our brains aren’t a precisely engineered composition of silicon, copper and lead, with microscopic transistor orchestrating mental operations. Our brains are composed of various classes of organic matter and compounds. I believe we can refer to a human’s 500 Internal Server Error as burnout.

I’m burned out.

I spent 2020 working. Working through the pandemic, through the riots in Portland, working through the signficant workforce reductions my employer imposed. I worked through weekends, through holidays, through everything. While the world tried to rip itself apart, my employer somehow mustered the power to pull my coworkers and I closer than ever. The volume of crises increased, so much so that even on the days I tried to take PTO (two, to be exact), I was pulled into triage calls and meetings.

Everything has suffered. My family is struggling to stay together. My dependents are starved for me. I’m starved for me. I just need to find the bug, the stack trace that cryptically says where everything went sideways. By understanding the failure, I will find the fix, and understand how to prevent this from occurring in the future. Without finding the fault, I’m at a loss.

This isn’t an intermittent failure on a vendor’s system. This is persistent and replicable.

I know posting something like this may compromise future opportunities. It may be too personal, or show too much of where I’m at right now for me to be appealing. That’s another funny thing about burnout; apathy. I don’t care. I haven’t posted since October. I need to write something. So why not write what I know so many of my colleagues are feeling. You fault me for being burnt out, I don’t want to work with you. Period. I own my choices. My position doesn’t accommodate me taking time – “we run lean.”

I am human. Not a cog in your machine.

Recovery

I watched this video and a lot of things clicked. I’ve been a complete asshole for months. Unreasonably irritable with the smallest triggers, depressed, anxious, uncommunicative. Hell, I don’t even really talk to my shrink. I’m paying $150/hr to only brush over surface shit with a stranger. A stranger a decade older than me that probably envies me for my income. Part of the generation that despises my generation. Productivity is.. comical.

I’m apothetic. I care enough to realize that I need to continue to meet baseline so I can support my way of life, but not enough to give any extra. They’ve taken enough from me, and I’m too tired to do anything more.

I’ve realized a change is no longer a feature request, but a requirement. It’s not a nice-to-have library. It’s stdio. I’m not sure if moving to another employer, where my personality will allow yet another manager to completely and totally exhaust my resources is the right choice, or pivoting to consulting is the way. However, running a business in my state? That’s a sure route to failure. Kernel panic.

My biggest regret and most missed aspect of doing what I do is building.

I just want to write code.

No clients. No tech support. No meetings. Just me, my IDE, and my headphones.

I’m tired. Happiness doesn’t exist at the bottom of the bottle, don’t ask how I learned that. Adaptogens are near useless when you 500, and talking to your superior (you know, the people that dictate whether you’re paying your bills next month) is a double edged sword. Asking for time off is always a gauntlet.

At this point, I’m at an impasse. If I don’t get five business days off without interruption, without being pulled into triage or some “must attend” meeting, then I don’t think I have a choice but to force a change. The future is unknown, but I know I cannot continue with what the past has shown me. One should not be dehumanized to the point of self-elimination by an employer.

I thought I was stronger than this.

In short, I don’t know what the road to recovery is. I don’t even know which direction is up at this point. What I do know is I have 20+ applications that haven’t been answered yet. I’m submitting more daily. And I’m pushing through with a smile taped on my face, and a kind tone in my voice. I just don’t know at this point.

500 Internal Server Error

Permission denied, please try again.

2020-10-20T10:23:00-07:00

Permission denied, please try again.

I’m at this odd stage in my career that I’m not sure how many people resonate with. I don’t have any developer friends, and most of my network is mostly comprised of security professionals who barely manage to write basic javascript, let alone understand the intricacies and elegance of OOP.

I’ve enough years experience to qualify for senior and lead level roles, yet I feel like over the past year my skill set has stagnated because I maintain a legacy codebase with very few new feature additions. The major work that I could do was completed late last year, and things have been running relatively smoothly since then. Any significant changes are for naught now as the codebase is being retired with the new year.

Despite that, I’ve applied for countless roles and interviewed for a handful of them. One such position seems to have all but evaporated (gotta love enthusiastic head-hunters that keep pushing the next round out), while the bulk of the others have either been straight denials, ghosts, or I’ve consciously pulled myself out of the pool simply because of blockers in the code challenges.

While talking with a coworker, I expressed this uncertainty — can’t quite call it a fear as there is only a perceived threat, but I can’t quite pin down the word for it otherwise. It almost feels like I made a mistake a year and some change ago, accepting my current role. While not entirely my fault for not realizing the gas light which was illuminating this new path for me, I take ownership for not reading between the lines enough.

On an average week I’m lucky to write code 10% of the time, and of that time, I’m converting Word documents to HTML — glad to say all those years of abstract OOP are being put to good use writing markup! I had a project for another department that my manager pulled me off of after 30 hours because of managerial politics. That project was subsequently outsourced.

I’m being shoe-horned into a support role that I don’t want anything to do with. Yet, every day I figure out how to better muscle my career in the direction I want to go through seeming insubordination, relationship building, and blatant disregard for things my manager wants me to focus on. Those things benefit management, but do nothing to push my career in the direction I want to go. Managers are supposed to help subordinates achieve their career goals, not railroad employees into roles they don’t want.

Regarding those interviews

One interview was for an “Application Security Engineer” role. That’s exactly the title I want, with the assumed responsibilities and work. Mind you, this company labels their pen testers with that title. They’re not doing appsec at all, as I came to find out during the interview. Sidenote: pen testing is only part of appsec, not all of it. Further, the interviewer seemed to be mocking me. Software developers don’t apply for infosec roles, he said. Only reason I got the interview was he was intrigued by my resume. The experience felt like a scene from Dinner with the Schmucks.

The next proved the power of networking does work, sometimes. This was for a role as a backend engineer using Ruby on Rails. That’s exactly the kind of role I’d love to pivot to at this point as PHP is great, but after so long it feels like I’ve become a pariah — PHP developers don’t know how to write code according to the industry. I guess this one proved that assumption well. The initial interview went well enough as I got to the coding challenge.

It went smoothly until it didn’t. I hit this one last blocker — something simple that I could easily do with PHP, but the ActiveRecord design pattern abstracted something seemingly simple in such a way I couldn’t figure out how to get it to cooperate. I just needed one value from a one model to appear in another model’s view. That’s it. I had exhausted the top end of the time limit they said it should take someone to do the challenge, and I wasn’t making anymore headway. I pulled myself out of the candidate pool, disheartened.

Finally, the last seemed like an excellent fit. Laravel backend, Vue frontend, at a vibrant startup with a ton of runway and a passionate team that cares about their mission. I care about their mission, too — it hits close to home. The first round was incredible, and we had to make a conscious effort to keep on time as the conversation was like chatting with an old friend. Then round two came. I was speaking with the lead developer and things seemed to moving great until he asked me what I’m passionate about.

When I said I’m passionate about security and would love the opportunity to mentor developers on writing secure code while positioning myself to lead security efforts in a thriving startup, he killed the interview. Said I should apply for those types roles instead and best of luck. Rejection for one’s passion isn’t a bad thing, is it? Doesn’t change the sting. That one hurt, like I’m doing something wrong as a developer.

Final thoughts

Based on this, I’ve tucked tail and decided being employed with some job security during economic downturn, a pandemic, and surely upsetting election year is better than dreaming of greener pastures for now. I’ve decided that if I want a Rails role, I need to build some stuff with it. Get really familiar with it. Take some time during my onslaught of meetings to actually write code. Maybe try to contribute to some open source projects so I can get a better understanding beyond to-do lists and recipe apps. Maybe do this until I hit the unofficial two year commitment I gave my manager and try the job hunt again.

Perhaps by then this phase in my career will feel less pubescent, and I’ll actually feel like the senior engineer I qualify for instead of the one that I feel like right now. I just have to find the key to unlock the gates which others keep, and get back to creating.

reformat

2020-03-29T14:12:00-07:00

reformat

In the midst of attempting to get Plex working again, I discovered that apparently 60GB of space was not adequate for the system to do what it needed to do. I had installed debian 10 a few months back after running on debian 9 for a few years. At the time, I had reasoned to leave the debian 9 install alone as well as continue to leave an old install of Windows 8.1 installed (For photoshop/illustrator/traktor), plus an even older Gentoo install. My thoughts were in case I ever had to boot up into Windows for design work or if I hadn’t backed up absolutely everything in the other envs, they’d still be there. Hi, my name is tj0 and I’m a data horder.

I decided that today would be a great day to go ahead and deal with this issue. Saying goodbye to windows, gentoo, and debian 9, and setting up debian 10 again, just with way more space. I figured I would document this process so that maybe someone else will be able to reference this for establishing their own environment.

kill it with fire

I’m running on the assumption you have a bootable USB ready. If not, download the latest ISO and run:

dd if=/home/tj0/debian.iso of=/dev/usb

Replace /home/tj0/debian.iso with the path to your downloaded ISO and /dev/usb with the path to your USB drive. I’m not covering this in depth as I assume you know what you’re doing.

Next, reboot your system and make sure your BIOS is set to boot from USB. Again, I’m assuming you know your own environment well enough to handle this yourself. I chose to use the graphical installer, for expedience. In my case, I selected the option to use the entire disk with separate /home, /tmp, and /var partitions with an encrypted LVM. This way the drive will be encrypted at rest, just in case anything warrants that necessity. Security is important, mkay.

Go grab a beer and find something to do, it’s going to take a while as the installer will overwrite the disk with junk, and a 1TB disk is going to take a while to get through this. In my case, I have a secondary system so I hopped on IRC and cracked open VS Code to write this. YMMV. Anticipate it taking a while, either way.

re-establishing your comfort zone

Ah, my least favorite part of all of this. Doing a fresh install is awesome because it provides you a green field to do whatever you want. But, when you’ve been doing development and security type work for long enough, you know what you like. In my case (like many others) I set up a public git repo with all my dotfiles so that at least it’s not a completely devastating process to get back to that sweet place of comfort.

This round I’m debating going with fluxbox again. I was running xfce, but it doesn’t offer the elegance and simplicity of a super lightweight window manager. While this is still up in the air, I’m also split on using i3, as it seems all the cool kids are using it now. I’m also a fan of tiling WM’s, although I haven’t used a true tiling WM in some time. The end goal is to have a fluid UX when switching between my work system (Mac) and my home system. While the terminal is the same (zsh ftw), it’s important that much of the behavior is the same. This reduces cognitive load and frustration when hacking at home vs at work. The biggest thing of all: it has to just work. I’m far past the age and time allowance where I can continuously tweak shit and fix breakages between updates (coughGentoo :(cough). I need this environment to behave as I expect, not be easily broken, and just work. Patience in this regard is thin.

As a general package list of what I require:

zsh
conky
tilda
VS Code
Android Studio
node
ruby
python3
php7.x
mariadb
postgres
redis
irssi
tmux
screen
vim
plexmediaserver
expressvpn
docker
virtualbox
gtk3
nmap
burp
wireshark
metasploit
jtr

One of my plans this round is to actually optimize my kernel for this environment. By default debian configures a very generalized kernel. While this doesn’t impact usability, as only modules are loaded as they are needed, it does lead to a bunch of extra cruft on your system that’s not needed. I haven’t noticed any significant performance impacts, but this round I want to make this system as stable as possible going forward. I’d also like to use the nvidia chip my system has versus the discrete intel graphics – this has historically been a real pain to get functioning properly as my board is an optimus-capable board and with the changing monitor configurations I have, shit gets complicated fast. In addition, it’d be nice to be able to use the CUDA functionality as I do a lot of CTF’s, and cracking with jtr could probably be a hell of a lot faster. It would also be nice to be able to use hashcat and similar, which require a dedicated GPU instead. I have no need for wine or any type of windows emulation; if I need to tread into that trash env I’ll spin up a VM.

covid

2020-03-25T18:16:00-07:00

covid

In December 2019, China experienced an outbreak of a new virus in the same class as SARS. Within a matter of months this virus has infected every country on earth, and projections indicate that hundreds of thousands are going to die in the best case scenario. Within the past two weeks, life in America has been completely changed. I’m sure you know all of this already as anyone that’s over the age of 15 is being bombarded with minute by minute news. I just wanted to take a moment, and write a post now that I have time to get one out.

finally fully remote

Since I started with my current employer I’ve wanted to be fully remote. Due to some arbitrary rule about me needing to be on-site three days a week that hasn’t ever come into full fruition, until this moment. I’m eight days into fully remote, and in these eight days I’ve achieved more than I have over the past two months working in office. I’ve saved $200 in parking and lunch fees; I’ve decreased my caffeine intake by several hundred milligrams a day; I’m even sleeping better. However, that means little due to some corporate decisions that have exponentially increased the stress regarding the future of my family.

I took a few hours yesterday, and finally took care of a big chunk of the cleaning I wanted to get done in my home office, and set up a more permanent working area for my work system beside my daily driver. Now if only I could figure out a way to use the same keyboard, and toggle between the two systems it would be perfect. I’ve also been focusing on learning more react native as I’m working on building a couple of apps which I’d love to see in a native format instead of the standard web apps I’ve built for the past decade.

setting guidelines

The hardest adjustment has been getting my children, and significant other to understand that when I’m in my office, I am to be left alone. I send a text notifying of meetings with the hope that distractions will be bountiful to keep the children quiet – this works 50% of the time. The normal distractions are no more significant than normal, albeit I’ve spent a bit more time on twitter because, well, that seems to be the only real-time feed of current information around this crisis coming directly from front-line workers and scientists.

I think the biggest thing with working remote is laying out a clear routine. Sure, I spend just as much time in my office at home as I did with my commute to the corp office, but I cut my work off at 17:00. I use the nex two hours daily to focus on my side projects. I have noticed that I’m not getting up as often as I should, so not taking breaks (not uncommon for me normally), and I’m forgetting to eat. Perhaps it’s the sheer level of stress this entire situation has everyone under right now, and I’m just not as quick to pick up the healthy, distractive habits as others.

full lockdown

As of an hour ago, Washington state is officially under a shelter-in-place order. This means you can be charged with a gross misdemeanor, and held liable for up to $1,000 in fines, and a year in jail. Now my hope is they truly enforce this. What I’ve seen over the past days is people are simply not taking this seriously, and may not until someone they love, or themselves, is directly affected by this virus. I’m not sure if it’s escapism, avoidance of grief, or the good ol American sense of rebellion, but people need to stay home. Period. We’re in this for the long haul, and many people have lost their jobs. If you’re one of those affected by job loss you have no reason to be driving around, burning fuel and spending money right now. Take your ass home, and stay there.

While I’m expected to still work 40 hours a week for the health of the company, I’m taking this opportunity to catch up on chores, do a very thorough spring cleaning, get a dump load ready, fill the garden with seedlings, and try to finally get some solid commits put into these side projects I’ve had running for months. Right now is a great opportunity for everyone to learn something new. Hell, get to know those you care about better than you ever have. The mortality rate of this illness is absolutely terrifying, and we will all be impacted both economically and physically by this. The psychological trauma is going to change us as a society. We need to make the best of this situation.

That being said, I do intend on monetizing one of the apps I’m working on. I’d love for it to grow into a business at some point, though we’ll see how the initial versions work out. A lot of people would say with the economy crashing, it’s a terrible idea to start up right now. I say fuck that. By the time this exits beta, the swell will be building, and I hope to ride that next wave up so never again will the wellbeing of my family be compromised by the control of some CEO that wouldn’t recognize me on the street.

this sucks

A lot of this experience has been a brutal emotional roller coaster. I’m witnessing a lot of people deal with grief for their first time. I’m (un)fortunate enough to have experienced witnessing my family die off as I grew up from a young age, so I’m all too familiar with these emotions. It doesn’t change the fact that the uncertainty of this situation is terrifying, gut wrenching, and surreal. The best thing about this is we will get through this, and never be the same. Hopefully for the better.

My hope is that American society finally sees where the federal government places their priorities, and votes for representatives that will serve the people. I hope that we as a society will replace the value we place in money into intelligence. I hope we come out of this being more conscientious about those around us, and how our actions have a ripple effect. We have this great opportunity, right now, to reset the status quo, and create a real future for America. No more of this bullshit red vs blue fighting, partisan politics that all end in zero sum. I may be delusional though. I wish I had a happy ending to this story, but as of right now, we’re still in the middle of it. Stay safe. Stay healthy. Hold those that matter to you close, but only after washing your goddamn hands. Thanks for reading.

backup

2020-03-03T10:50:00-08:00

backup

This post will probably be pretty short. I wanted to document this process flow so I could potentially script it in the future, and apply it other than in the instance that has led to this being necessary. I’ve been put in a position where I need to create backups and perform modifications on core system files for some high volume servers. While we could do the deed in production (who doesn’t like breaking prod?), it’s not the best idea given the revenue losses that would be suffered should downtime occur. This has led to a bit more discovery in the power of ssh, dd, pv, and qemu. Let’s get started.

Note: Names, IPs, ports, etc have been changed because reasons.

I need teh remote codez, plz

One of the bigger hurdles is how the servers are configured. They’re locked down pretty tight, including good UAC practices and some rather…. immature SSH configurations, albeit secure. This presented a bit of an issue initially, as key files are password protected, then sudo accounts aren’t exactly relaxed. I understand the reasoning for this, but it hasn’t contributed to making these efforts any easier. Anyways, we craft the initial ssh command as so:

> ssh -p 4321 sudoUser@192.168.100.12 -i keyfile.pem
Enter passphrase for key 'keyfile.pem':
Last login: Mon Mar  2 18:10:27 2020 from 10.0.11.2
sudoUser@prodserver1:~$

That gives us the basic access. But, how could you possible pull a raw image of this disk via this tunnel? You certainly cannot run dd and pipe the output back through the line; that’s not how this tunnel is configured. Why would we use dd instead of generating a tarball instead? Well, there are a few reasons for that, my biggest reason being I want to make sure I have a near identical copy of the entire system. Running sudo tar -cvzf prodserver1.tgz /* won’t provide us an exact replica, and will only copy the filesystem files. I want all the bytes, not just the data, so we use dd instead. Craft the dd command as so:

> dd if=/dev/xvda of=/home/tj0/prodserver1.img

The nice thing about dd is the fact that the images are raw binary data. This makes the tool useful for all kinds of things, including generating bootable USB media (with some additional parameters). Anyways, we now have our tunnel, and we have our method of collecting all the data. Why’d I mention pv you ask. Well, dd doesn’t say much – it’s a utility of few words. Essentially you’ll be sitting there, waiting for something, anything. You could run ls -alh | grep prodserver1.img and see what the file size is, but that’s manual, and it doesn’t provide the satisfaction of progress bars. pv provides us feedback.

Crafting the query

At this point, we have pretty much all we need so let’s throw it all together and get that image:

> ssh -p 4321 -tt sudoUser@192.168.100.12 -i keyfile.pem "sudo dd if=/dev/xvda " | pv | dd of=/home/tj0/prodserver1.img
Enter passphrase for key 'keyfile.pem': <enter keyfile password>
<enter sudo password>
0.00 B 0:00:06 [0.00 B/s] [<=>                                          ]
9.63GiB 0:24:52 [6.05MiB/s] [<=>                                          ]

You’ll see pv dump some output. Don’t let this scare you. In my case, I had to enter the sudo user password as well, otherwise sudo (which you cannot see the output from) will sit there. Waiting. Leaving you with a 0B file, and questioning if you are ever going to be a great terminal wizard. Just enter the password. You might notice that I added -tt to the ssh command. This is required because dd requires some type of TTY being available, and -tt forces a pseudo-tty to be available for dd to bind to, or something like that. RTFM for exact technical details.

Getting it VirtualBox ready

This step is interim, and possibly won’t be necessary for you. Given I’m doing this on a Mac and I use VirtualBox in other projects (and Vagrant, ftw), This is a simple method of achieving the end goal:

VBoxManage convertfromraw prodserver1.img vmdkprodserver1.vmdk --format VMDK

Make sure to run VBoxManage convertfromraw and check out the additional options. There’s a VHD option, which is the format needed for Azure according to this document.

That’s it for now

YMMV. This was a rather hackish, albeit necessary process that is being applied across several servers in an effort to keep several teams from winding up in a really, really crappy situation. Ping me if you need guidance or have input of better methods to doing this.

prep

2020-02-25T10:50:00-08:00

prep

The time has come to dust off those old study materials, and get re-acquainted with some long forgotten concepts. The only issue is I don’t have old study materials – I’m self taught, so it’s Google-fu I’m using this time around. An added benefit this round is I’ll document what I’m learning here, which will do two things: (1) help the non-existent readers of this fine blog and (2) generate content! Which is something I really badly need to do (it’s been 21 days since I wrote dmidecode). Let’s do this.

Find the known unknowns

I don’t miss the days of being junior. Just getting the attention of recruiters and hiring managers anywhere hiring junior level was incredibly difficult, and a lot of it had to do with a ton of unknown unknowns. That being said, you only come to know what you don’t know through experience (and some brutal code reviews). At this point in my career, and given the fact I’ve worked predominantly with PHP (don’t do this – learn something actually marketable, and “professional”), I’m starkly aware of the fact my knowledge of data structures and algorithms is about as strong as my pinky toe.

It’s time to change that. I’ve worked in great places, and I’ve worked in really, really shitty places, and frankly I’m over the hot-cold nature of this industry. As much as I detest the idea of technical interviews and the concept that “a real professional needs to know these theories like they’re their gospel”, the powers that be require strong knowledge in these areas, so to cite Coding Interview for Dummies:

In reality, being aware of existing data structures and selecting the appropriate ones to tackle the problem at hand is more important than knowing the intricate implementation details.

Basically, my objective is pretty simple right now. Learn some shit, write some shit, and apply some shit. Seems pretty simple, so this post will serve as a roadmap of sorts going forward. I’ll tackle some definitions and get some high level explanations compiled, and will likely expound on this post a bit in the coming weeks as things come into fruition. Please note that some of the code was copied (citations are all at the bottom of this page). Some I wrote myself for the sake of this article. Consider this to be a compilation of a bunch of resources, condense into bite-sized chunks for your easy consumption.

Data structures

What are data structures.. Well, we use them to organize data, which helps enable us to use the data more effectively. Oftentimes, junior developers (especially self taught ones) learn just enough about arrays to brute force their way into a really shitty Wordpress install, and then have people screaming at them because their site is slower than a 56k modem connection trying to download some pixelated, oversized 90s album art.

By utilizing data structures, we are able to optimize our resource consumption, while maximizing the throughput of data over a given time period. ~~Basically, we use them to play Einstein, in a really remotely, not really related kind of way.~~ Data structures do have a lot of to do with our use of computational space and time. Here’s the major ones.

Linked List

Linear collection of data elements (nodes), each pointing to the next node by means of a pointer. Linked lists consist of a group of nodes and represent a sequence; there are three types:

Single-linked list: each node points to the next node, and the final points to null Double-linked list: each node has two pointers [P and N], where P points to the previous node and N points to the next node. The last node’s N points to null. Circular-linked list: each node points to the next, and the last node points back to the first node.

# Python

class DoublyNode:
    def __init__(self, val):
        self.val = data
        self.next = None
        self.prev = None

    def traverse(self):
        node = self # start from the head node
        while node != None:
            print node.val # access the node value
            node = node.next # move on to the next node

// PHP

$list = new SplDoublyLinkedList();
$list->push('a');
$list->push('3');
$list->push('v');
$list->push('1');
$list->push('p');
$list->rewind();

while ($list->valid()){
    echo $list->current() . PHP_EOL;
    $list->next();
}

When and why:

Linked lists save memory by only allocating the memory required for the values to be stored. In addition, nodes can live anywhere in memory as opposed to an array, where indexes live sequentially. However, this comes at the cost of look up time which is linear due to how linked lists are constructed. Each node in the chain must be checked, one at a time, for a matching value.

Time Complexity:
  Access: O(n)
  Search: O(n)
  Insert: O(1)
  Remove: O(1)

Stack

A collection of elements with two principal operations: push (add/append) and pop (remove). Stacks are LIFO (last in, first out) data structures.

# Python

stack = ['First', 'Second', 'Third']
stack.append('Fourth')
stack.append('Fifth')
print(stack)
print(stack.pop())
print(stack)
print(stack.pop())
print(stack)

# Output:
['First', 'Second', 'Third', 'Fourth', 'Fifth']
Fifth
['First', 'Second', 'Third', 'Fourth']
Fourth
['First', 'Second', 'Third']

// PHP
$stack = ['First', 'Second', 'Third'];
array_push('Fourth');
array_push('Fifth');

var_dump($stack);
array_pop($stack);
var_dump($stack);
array_pop($stack);

// Output:
array(5) {
  [0] =>
  string(5) "First"
  [1] =>
  string(6) "Second"
  [2] =>
  string(5) "Third"
  [3] =>
  string(6) "Fourth"
  [4] =>
  string(5) "Fifth"
}
array(4) {
  [0] =>
  string(5) "First"
  [1] =>
  string(6) "Second"
  [2] =>
  string(5) "Third"
  [3] =>
  string(6) "Fourth"
}
array(3) {
  [0] =>
  string(5) "First"
  [1] =>
  string(6) "Second"
  [2] =>
  string(5) "Third"
}

Note: As of PHP 7, PHP now offers various data structures as an alternative to arrays. See requirements and installation for more information.

When and why:

Stacks are useful for back tracing access to previous elements or operations and matching recursive elements or operations. Consider the Undo functionality in your code editor – stacks work very simliarly to that (and that’s where the usefulness comes in).

Time Complexity:
  Access: O(n)
  Search: O(n)
  Insert: O(1)
  Remove: O(1)

Queue

A collection of elements supporting two operations, enqueue and dequeue. Queues are FIFO (first in, first out) data structures. You can think of a queue as a pipeline, or single lane road, where the first item to enter is the first to leave, ad infinitum.

# Python

from collections import deque
queue = deque(['Fourth', 'Third', 'Second', 'First'])
print(queue)
queue.append('Fifth')
print(queue)
queue.append('Sixth')
print(queue)
print(queue.popleft())
print(queue.popleft())
print(queue)

# Output
deque(['Fourth', 'Third', 'Second', 'First'])
deque(['Fourth', 'Third', 'Second', 'First', 'Fifth'])
deque(['Fourth', 'Third', 'Second', 'First', 'Fifth', 'Sixth'])
Fourth
Third
deque(['Second', 'First', 'Fifth', 'Sixth'])

When and why:

Queues are useful when you want to process one thing at a time. For example, we have a cron job that runs nightly which is responsible for enabling and disabling user accounts in various platforms. A queue is used in this instance as we don’t want to bomb the mail server with 200 requests to send emails out and we also don’t want to block resources during larger batches. By using a queue, we sequentially run through the affected accounts, and everything runs smoothly.

Time Complexity:
  Access: O(n)
  Search: O(n)
  Insert: O(1)
  Remove: O(1)

Tree

A Tree is an undirected, connected, acyclic graph. See All you need to know about tree data structures for a much more thorough explanation than I will provide at this time.

There is some terminology that needs to be understood when discussing trees:

Root is the very first, or root, node in a tree
Edge is a link between two nodes
Child is a node that has a parent node
Parent is a node that has nodes underneath it
Leaf is a node that has no child nodes
Height is the longest path in the tree
Depth is the distance from a node to the root node

Binary Tree

A Binary Tree is a tree data structure in which each node has at most two children, which are referred to as the left child and right child. This is not to be confused with a binary search tree, which is slightly different in the regard that any value in each node must be greater than or equal to the value stored in the left child node, while also being less than or equal to any value stored in the right child node. Confusing, yes, but we’ll get to that momentarily.

# Python

class BinaryTree:
    def __init__(self, value):
        self.value = value
        self.left_child = None
        self.right_child = None

    def insert_left(self, value):
        if self.left_child == None:
            self.left_child = BinaryTree(value)
        else:
            new_node = BinaryTree(value)
            new_node.left_child = self.left_child
            self.left_child = new_node

    def insert_right(self, value):
        if self.right_child == None:
            self.right_child = BinaryTree(value)
        else:
            new_node = BinaryTree(value)
            new_node.right_child = self.right_child
            self.right_child = new_node

top = BinaryTree('root')
top.insert_left('child_a')
top.insert_right('child_b')
child_a = top.left_child
child_b = top.right_child
child_a.insert_left('child_c')
child_a.insert_right('child_d')
child_b.insert_left('child_e')
child_a.insert_right('child_f')

# Structure
    root
    /\
  a    b
 / \  / \
c  d  e  f

An important detail about trees is the flexiblity for traversing a tree. We have two methods available, Depth-First Search (DFS) or Breadth-First Search (BFS). Let’s cover various methods of traversal now, then I’ll discuss when and why to use trees.

Depth-First Search follows a path all the way to the final node before backtracking and navigating another path. The algorithm would go something like:

Start at root
Go to child_a; print it.
Go to child_c; print it. Since this node doesn’t have any children, backtrack to child_a.
Go to child_d; print it. Since this node doesn’t have children, backtrack to root as all children of child_a have been enumerated.
Go to child_b; print it.
Go to child_e; print it. Since this node doesn’t have any children, backtrack to child_b.
Go to child_f; print it. Since this node doesn’t have children, and is the last node in the tree, finalize execution.

Now that we’re familiar with the general algorithm, we can discuss a few types of DFS.

Pre-order: Exactly what we did in the algorithm above. Prints the value of a node. If, and only if it has a left child, goes to the left child, and prints it. Repeats for the right child and subsequent nodes. Output looks like root-a-c-d-b-e-f.

def pre_order(self):
    print(self.value)

    if self.left_child:
        self.left_child.pre_order()
    if self.right_child:
        self.right_child.pre_order()

In-order: this runs left first, middle second, and right last. Using the tree above as an example, the resulting output of this style would be c-a-d-root-e-b-f.

def in_order(self):
    if self.left_child:
        self.left_child.in_order()

    print(self.value)

    if self.right_child:
        self.right_child.in_order()

Post-order: this runs bottom up, consuming each level of nodes as it works its way up the tree. Output would look like c-d-a-e-f-b-root

def post_order(self):
    if self.left_child:
        self.left_child.post_order()

    if self.right_child:
        self.right_child.post_order()

    print(self.value)

Breadth-First Search traverses by level and depth. A rudimentary illustration of what this means, using the same tree we used in our previous example, would look something like this:

    root        <-- Level 0
    /\
  a    b        <-- Level 1
 / \  / \
c  d  e  f      <-- Level 2

def breadthfirst(self):
    queue = Queue()
    queue.put(self)

    while not queue.empty():
        current_node = queue.get()
        print(current_node.value)

        if current_node.left_child:
            queue.put(current_node.left_child)

        if current_node.right_child:
            queue.put(current_node.right_child)

# Output
root-a-b-c-d-e-f

Binary Search Tree

As I mentioned earlier, we’d get to this. The most important distinction you need to remember is that the value of a node is greater than the value of its left child, but less than the value of its right child. Like this, right_child > root > left_child, or like:

And implemented:

def BinarySearchTree:
    def __init__(self, value):
        self.value = value
        self.left_child = None
        self.right_child = None

    def insert_node(self, value):
        if value <= self.value and self.left_child:
            self.left_child.insert_node(value)
        elif value <= self.value:
            self.left_child = BinarySearchTree(value)
        elif value > self.value and self.right_child:
            self.right_child.insert_node(value)
        else:
            self.right_child = BinarySearchTree(value)

    def find_node(self, value):
        if value < self.value and self.left_child:
            return self.left_child.find_node(value)
        if value > self.value and self.right_child:
            return self.right_child.find_node(value)

        return value == self.value

When and why:

The main advantage for using a binary search tree is it extends functionality of a normal array. While arrays are constant time for reads, adding an index at P position will result in any indices after P to be re-indexed, which is a linear time operation (O(n)). This operation isn’t terribly slow, but BST can perform better. They perform better because instead of relying on indexes for reference to values, they rely on their implicit structure. As a result, insertions and deletions are done at logarithmic time (O(log n)).

One of the main use cases for this data structure is indexing database entries. For example, if you have 1 million entries, and you need to find a specific field, you can locate this user in at most 21 iterations. Compare this to iterating over an array, which would require at minimum 20,000 iterations. Quite a big difference.

Time Complexity:
  Access: O(log(n))
  Search: O(log(n))
  Insert: O(log(n))
  Remove: O(log(n))

Trie

A trie, sometimes called a radix or prefix tree, is a kind of search tree that is used to store a dynamic set or associative array where the keys are usually Strings. No node in the tree stores the key associated with that node. A node’s position in the tree defines the key with which it is associated. All the descendants of a node have a common prefix of the String associated with that node, and the root is associated with the empty String.

Python implementation

PHP implementation

When and why:

Typically tries are used when doing lookups, such as searching for a specific word or string of words within a body of text. Other use cases include when you have a fixed dictionary that you want to perform lookups aginst quickly. When compared to a hashtable, it may require less storage but may also take longer to look up.

Fenwick Tree

A Fenwick tree, sometimes called a binary indexed tree, is implemented as an implicit data structure using an array. Given an index in the array representing a vertex, the index of a vertex’s parent or child is calculated through bitwise operations on the binary representation of its index. Each element of the array contains the pre-calculated sum of a range of values, and by combining that sum with additional ranges encountered during an upward traversal to the root, the prefix sum is calculated.

This stack exchange thread has some excellent discussion about Fenwick trees, and I highly recommend reviewing it for more details than I can provide in regards to this data structure.

# Python, see link [10]

class Fenwick():
    def update(self, idx, val):
        while idx <= self.MaxVal:
            self.tree[idx] += val
            idx += idx & (-idx)

    def query(self, idx):
        total = 0
        while idx > 0:
            total += self.tree[idx]
            idx -= idx & (-idx) # subtract the least significant, non-zero bit from idx
        return total

    def __init__(self, size = []):
        self.MaxVal = len(size)
        self.tree = [0 for i in xrange(self.MaxVal)]
        for idx in xrange(0, self.MaxVal):
            self.update(idx, size[idx -1])

// PHP

class Fenwick
{
    public $max;
    public $tree;

    public function __construct(array $data = [])
    {
        $this->max = sizeof($data);
        echo "Max: " . $this->max . PHP_EOL;
        for ($i = 0; $i < $this->max; $i++) {
            $this->tree[$i] = $data[$i];
        }

        echo "Tree: " . print_r($this->tree, true) . PHP_EOL;
        for ($j = 0; $j < $this->max; ++$j) {
            $this->update($j, $data[$j]);
        }
    }

    /**
     * Don't rely on this method -- I'm not sure it's working
     * The while loop was causing an infinite loop, so I removed it.
     *
     * I believe this was caused by the bitwise operation.
     */
    public function update(int $idx, int $val): void
    {
        // while ($idx <= $this->max) {
            $this->tree[$idx] += $val;
            $idx += ($idx & (-$idx));
        // }
    }

    public function query(int $idx): int
    {
        $total = 0;

        while ($idx > 0) {
            $total += $this->tree[$idx];
            $idx -= ($idx & (-$idx));
        }
        return $total;
    }
}

$t = new Fenwick(['0', '15', '325', '6', '11', '17']);
echo 'Result: ' . $t->query(3) . PHP_EOL;
exit;

// Output:
Max: 6
Tree: Array
(
    [0] => 0
    [1] => 15
    [2] => 325
    [3] => 6
    [4] => 11
    [5] => 17
)

Result: 662

When and why:

Fenwick trees, or binary indexed trees, are most useful when calculating the sum of a range. They’re also commonly used in coding challenges. I personally have not found a practical use for this data structure in my day to day tasks.

Time Complexity:
  Range Sum: O(log(n))
  Update: O(log(n))

Segment Tree

A Segment tree, is a tree data structure for storing intervals, or segments. It allows querying which of the stored segments contain a given point. A node may store one or more data members of an interval, which may be queried later.

# Python, see link [12]

class SegmentTree:
    def __init__(self):
        self.tree = None

    def buildTree(self,List):
        n = len(List)
        self.tree = [0 for i in range(2*n)]

        # Initialize leaf nodes
        for i in range(n):
            self.tree[i+n] = List[i]

        # Calculate parents and build the tree
        for i in range(n-1,0,-1):
            self.tree[i] = self.tree[i<<1] + self.tree[i<<1 | 1]

    def updateInRange(self, value, left, right):
        n = len(self.tree)//2

        for i in range(left+n, right+n+1):
            # Update the leaf node
            self.tree[i] += value

            j = i
            # Move upwards and update parents
            while j>1:
                self.tree[j>>1] = self.tree[j] + self.tree[j^1]
                j >>= 1

    def sumInRange(self, left, right):
        n = len(self.tree)//2
        result = 0
        left += n
        right += n

        # Add the current node to the sum if its parent is not included in the same
        # (resulting in faster computation of result)
        while left<right:
            # If left is odd then it means that it is the right child of it's parent
            # and the interval includes only left and not it's parent. So, include
            # this node to sum and move to the parent of it's next node.
            # Similar condition is applied to right index.
            if left&1:
                result += self.tree[left]
                left += 1
            if right&1:
                right -= 1
                result += self.tree[right]

            left >>= 1
            right >>= 1

        return result


def main():
    segmentTree = SegmentTree()

    #Sample test case
    sampleList = [1,3,5,7,9,11]
    segmentTree.buildTree(sampleList)

    print("Array representation of tree :",segmentTree.tree)
    print("Sum of elements in the interval [1,3] :",segmentTree.sumInRange(1,4))

if __name__ == "__main__":
    main()

When and why:

Used as a more versatile, heavier handed version of FT/BIT. I need to do further research on this data structure as well to fully comprehend what its valid use cases are.

Time Complexity:
  Range Query: O(log(n))
  Update: O(log(n))

Heap

A Heap is a specialized tree-based data structure that satisfies the heap property. If $a is a parent node of $b, then the key (the value) of $a is ordered with respect to the key of node $b with the same ordering applying across the entire heap. There can be two types of heap:

Max heap: the keys of parent nodes are always greater than or equal to those of the children and the highest key is in the root node

// PHP

$heap = new SplMaxHeap();
$heap->insert(65);
$heap->insert(13);
$heap->insert(94);
$heap->insert(0);

foreach($heap as $number) {
    echo $number . PHP_EOL;
}

// Output
94
65
13
0

Min heap: the keys of parent nodes are less than or equal to those of the children and the lowest key is in the root node

# Python

import heapq

heap = [12, 45, 9, 1, 53, 8, 27, 66, 20]

heapq.heapify(heap)
print(heap)

# Output
[1, 8, 9, 12, 20, 27, 45, 53, 66]

// PHP

$heap = new SplMinHeap();
$heap->insert([22,333]);
$heap->insert([2,33]);
$heap->insert([222,3]);

var_dump($heap->extract());
var_dump($heap->extract());
var_dump($heap->extract());

// Output
array ( 0 => 2, 1 => 33, )
array ( 0 => 22, 1 => 333, )
array ( 0 => 222, 1 => 3, )

When and why:

Useful whenever you need quick access to the largest or smallest item in a set. The reason for this is the item you need will always be the first in the array, or at the root of the tree. Insertions are fast, and this makes heaps a good way of dealing incoming events or data and always having access to the earliest or biggest indexes. Also valuable for queues, schedulers, or anything else where the earliest item is desired.

Time Complexity:
  Access Max / Min: O(1)
  Insert: O(log(n))
  Remove Max / Min: O(log(n))

Hashing

Hashing is used to map data of an arbitrary size to data of a fixed size. If two hash keys map to the same value, a collision occurs. Sometimes, hashes can be used in what are call hash tables, which are nothing more than a structure that can map keys to values, for example:

// PHP
return [
    'key' => 'value',
    'foo' => 'bar'
];

# Python

# Shorthand
hashmap = {'key': 'value', 'foo': 'bar'}

# Using dict()
hashmap = dict({'key': 'value', 'foo': 'bar'})

return hashmap

To facilitate collision resolution we have a couple of tools available to us:

Separate Chaining: in separate, independent buckets, each bucket contains a list of entries for each index. The time for hash map operations is the time to find the bucket (constant time), plus the time to iterate through the list Open Addressing: when a new entry is inserted into a bucket, the buckets are examined (starting with the hashed-to-slot and proceeding in some sequence) until an unoccupied slot is found. This is called open addressing because sometimes the hash does not point to the location of the value.

Note: Hashmaps resembles JSON syntax for a reason. They’re technically the same, and can also be referred to as associative arrays, depending on the language.

When and why:

Hashing is useful in a lot of instances. One use case would be an unordered list of cities that you want to index by state so you can use JavaScript to dynamically populate some drop downs in a form. You could use the user’s state selection as the key lookup in the hash, and this would ensure you have a one to many relationship between your state and city names. Really, hashes are incredibly useful far beyond such a small use case. As the note says, hashmaps are identical to JSON, and JSON has definitely taken off as the dominant structure of API communications in the past decade. That didn’t happen just by chance.

Graph

A Graph is a data structure consisting of a finite (and possibly mutable) set of nodes, or points, together with a set of unordered pairs of these vertices (for an undirected graph) or a set of of ordered pairs (for a directed graph). These pairs are known as edges, and for a directed graph can be referred to as arrows. See this wikipedia for more information, because I’m still super fuzzy on these as I’ve never had a reason to implement this before.

// PHP

// This is a Dijkstra implementation, see link [14]
class Graph
{
  protected $graph;

  public function __construct($graph) {
    $this->graph = $graph;
  }

  public function shortestPath($source, $target) {
    // array of best estimates of shortest path to each
    // vertex
    $d = array();
    // array of predecessors for each vertex
    $pi = array();
    // queue of all unoptimized vertices
    $Q = new SplPriorityQueue();

    foreach ($this->graph as $v => $adj) {
      $d[$v] = INF; // set initial distance to "infinity"
      $pi[$v] = null; // no known predecessors yet
      foreach ($adj as $w => $cost) {
        // use the edge cost as the priority
        $Q->insert($w, $cost);
      }
    }

    // initial distance at source is 0
    $d[$source] = 0;

    while (!$Q->isEmpty()) {
      // extract min cost
      $u = $Q->extract();
      if (!empty($this->graph[$u])) {
        // "relax" each adjacent vertex
        foreach ($this->graph[$u] as $v => $cost) {
          // alternate route length to adjacent neighbor
          $alt = $d[$u] + $cost;
          // if alternate route is shorter
          if ($alt < $d[$v]) {
            $d[$v] = $alt; // update minimum length to vertex
            $pi[$v] = $u;  // add neighbor to predecessors
                           //  for vertex
          }
        }
      }
    }

    // we can now find the shortest path using reverse
    // iteration
    $S = new SplStack(); // shortest path with a stack
    $u = $target;
    $dist = 0;
    // traverse from target to source
    while (isset($pi[$u]) && $pi[$u]) {
      $S->push($u);
      $dist += $this->graph[$u][$pi[$u]]; // add distance to predecessor
      $u = $pi[$u];
    }

    // stack will be empty if there is no route back
    if ($S->isEmpty()) {
      echo "No route from $source to $targetn";
    }
    else {
      // add the source node and print the path in reverse
      // (LIFO) order
      $S->push($source);
      echo "$dist:";
      $sep = '';
      foreach ($S as $v) {
        echo $sep, $v;
        $sep = '->';
      }
      echo "n";
    }
  }
}

When and why:

A lot of the big companies are offering graph APIs now, and there is a lot of excitement around graphs. This is another area I’m weak in and need to spend some time studying and figuring out what the application of this data structure is. Suppose that just means more content needs to be written.

Algorithms

So, I was gonna write about some algorithms at this point. As I started to dig into them (and it being quite late in the day, post work and pre-beer), I’ve decided that the algorithms portion will probably be a multipart series. There are simply too many (a few small ones are even implemented in this post) to be able to cram them all into this one-stop-learn-everything-you-need-so-one-of-the-Big-Four-will-hire-you post. However, what I will leave you with are some additional questions and resources that will surely keep you busy while I take another month to write something new. If you made it here, thank you. If not, well, you probably were never here to begin with.

Questions to ask

How big is the size of the input?
How big is the range of values?
What kind of values are there? Are there negative numbers? Floating points? Will there be empty inputs?
Are there duplicates within the input?
What are some extreme cases of the input?
How is the input stored? If you are given a dictionary of words, is it a list of strings or a trie?

Links:

1. Awesome Algorithms - Online Courses 2. Coding Interviews for Dummies 3. Time Complexity 4. 10 Most Popular Coding Challenge Websites (2016) 5. Short Circuit Evaluation 6. Tech Interview Handbook 7. Base CS 8. Dynamic Programming 9. Linked Lists 10. Fenwick Trees 11. Binary Indexed Tree 12. Segment Tree Sum of a given range 13. Segment Tree (Wikipedia) 14. Data Structures 4 15. All you need to know about tree data structures 16. How to implement a binary tree 17. Why Use Binary Search Tree 18. Binary Indexed Trees 19. A simple approach to segment trees

visudo

2019-12-31T14:30:00-08:00

visudo

If you’re here, thanks for keeping up. It’s been a bit since I updated this, and this is kind of a part two in regards to a side project and the wonderful issues I’ve been having to overcome as a result of big ambitions and little time. If you’re not caught up, check out touch remote dev so you have some context before continuing.

Christmas break didn’t bring many opportunities to sit down and hack on anything, but I did get one day – a day was all that was needed apparently. This day afforded me the ability to sit down and dig into the underlying issue I was experiencing with tenancy/multi-tenant. The problem was pretty straight forward, www-data was not allowed to execute /usr/sbin/nginx configtext. Well, duh! nginx is only supposed to be executed by root and it spawns child processes under the specified user parameter.

Google, you suck

Now I knew one of the fundamental issues I was having when I initially embarked on this journey months ago – it didn’t have to do with ACLs or permissions. It had to do with a basic system property that is in place to ensure no one mischevious does anything vile. So, how is one to fix this problem? Well, what tenancy won’t mention in their documentation (literally, this isn’t mentioned anywhere) is you have to modify src/config/webserver.php to look something like this:

<?php

/*
 * This file is part of the hyn/multi-tenant package.
 *
 * (c) Daniël Klabbers <daniel@klabbers.email>
 *
 * For the full copyright and license information, please view the LICENSE
 * file that was distributed with this source code.
 *
 * @see https://tenancy.dev
 * @see https://github.com/hyn/multi-tenant
 */

return [
    /**
     * Redacted some apache stuff...
     */

    /**
     * Nginx webserver support.
     *
     * @see http://nginx.org
     */
    'nginx' => [
        /**
         * Whether the integration with nginx is currently active.
         */
        'enabled' => true,

        /**
         * The php sock to be used.
         */
        'php-sock' => 'unix:/var/run/php/php7.3-fpm.sock',

        /**
         * Define the ports of your nginx service.
         */
        'ports' => [
            /**
             * HTTP, non-SSL port.
             *
             * @default 80
             */
            'http' => 80,
            /**
             * HTTPS, SSL port.
             *
             * @default 443
             */
            'https' => 443
        ],

        /**
         * The generator taking care of hooking into the nginx services and files.
         */
        'generator' => \Hyn\Tenancy\Generators\Webserver\Vhost\NginxGenerator::class,

        /**
         * The view that holds the vhost configuration template.
         */
        'view' => 'tenancy.generators::webserver.nginx.vhost',

        /**
         * Specify the disk you configured in the filesystems.php file where to store
         * the tenant vhost configuration files.
         *
         * @info If not set, will revert to the default filesystem.
         */
        'disk' => null,

        'paths' => [

            /**
             * Location where vhost configuration files can be found.
             */
            'vhost-files' => [
                '/ABSOLUTE/PATH/TO/YOUR/PROJECT/src/app/tenancy/webserver/nginx'
            ],

            /**
             * Actions to run to work with the Nginx service.
             */
            'actions' => [
                /**
                 * Action that asserts nginx is installed.
                 */
                'exists' => '/etc/init.d/nginx',
                /**
                 * Action to run to test the nginx configuration.
                 *
                 * @info set to a boolean to force the response of the test command.
                 *  true succeeds, false fails
                 */
                'test-config' => 'sudo /etc/init.d/nginx configtest',
                /**
                 * Action to run to reload the nginx service.
                 *
                 * @info set to null to disable reloading.
                 */
                'reload' => 'sudo /etc/init.d/nginx reload'
            ]
        ]
    ]
];

There are a couple things here that tenancy doesn’t mention in their docs, at least not where I could find. The first is the fact that 'enabled' isn’t controlled by an environment variable and is set to false to begin with. Toggle that to true, if you’re using nginx. If you’re using apache, you’re SOL, I can’t help you. Next, the important lines are the ones regarding 'test-config' and 'reload'. These are commands that tenancy will call, when a new user attempts to register on your platform, and if www-data cannot run those commands, tenancy fails in all kinds of weird and frustrating ways. We’re not done yet, though. Just because you slap sudo on the front of those commands doesn’t mean shit yet. www-data needs to be added to /etc/sudoers if this is to work.

$ sudo visudo

#
# This file MUST be edited with the 'visudo' command as root.
#
# Please consider adding local content in /etc/sudoers.d/ instead of
# directly modifying this file.
#
# See the man page for details on how to write a sudoers file.
#
Defaults        env_reset
Defaults        mail_badpass
Defaults        secure_path="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"

# Host alias specification

# User alias specification

# Cmnd alias specification

# User privilege specification
root    ALL=(ALL:ALL) ALL

# Allow members of group sudo to execute any command
%sudo   ALL=(ALL:ALL) ALL

#www-data ALL=(ALL:ALL) NOPASSWD: /etc/init.d/nginx configtest, /usr/sbin/service nginx configtest, /usr/sbin/service nginx reload
www-data ALL=(ALL:ALL) NOPASSWD: ALL

# See sudoers(5) for more information on "#include" directives:

#includedir /etc/sudoers.d

Now, you can see there are a couple additions in there over your default configuration. I will start by saying this, regarding those modifications: the modification I am using is insecure and needs revision. Given www-data is the webserver user, which PHP is running as, this user should be heavily restricted, especially given the context we’re working in right here. Should a user figure out how to inject a command, and it be executed by www-data, all they’d have to do is add sudo and the server would be totally pwned.

Either way, I was receiving some other dumb error because sudo kept spewing it’s “great power comes great responsibility” message, and I’ve forgotten how to shut that stupid thing up. As soon as I completed this step, registration started working. With due time, I’ll get back to that, but for now. It finally fucking works.

Then I turned around and broke the shit out of it by trying to convert the entire auth flow to vue.

Oh well, it’s been a learning experience. I’m going to scrap everything again and this time, as I follow the chain of random tutorials and guides, and docs, I’m going to document the hell out of my workflow and dump it all here. Maybe that’ll be my New Year’s Day task – give people an actively correct guide to getting a base tenancy system set up and working. Till next time.

mkdir

2019-12-31T14:30:00-08:00

mkdir

Given my current work situation, I feel incredibly isolated and know that when I move on to my next venture, the imposter syndrome I worked out at my last gig will come back in spades. As a result, I’ve been thinking about this idea for a while. There are platforms that already do something similar, but they aren’t dedicated to just this idea, and oftentimes have this reputation of being unfriendly towards new comers (cough StackExchange cough). As soon as you put up a post, someone comes in and offers the incredibly deep anidote of “you’re wrong, incapable, and shouldn’t do it this way. Oh, and this is a duplicate. clicks downvote without explanation.

This idea is simple. A network of developers that can post code (even entire projects) and get reviews. Yes, you can do this on github, but that only works if you’re working on open source software, and if others take interest in your project. But what if you’re in my situation? I’m working on completely rewriting a legacy codebase to make it somewhat readable and maintainable, and I’m alone in this venture. In a company of 3,000+ people, I am the only PHP “expert.” So that means code reviews are non-existent, I have no one to pass ideas around with, and I’m constantly hitting my head against a wall when I run into something that I can’t even imagine a day-two junior dev writing.

I’m not sure what I’d call this platform, but if it existed, at this point in my career I’d lean on it heavily. There’s no reason that a developer should be left on an island and trust that their work is the best it can be when they are their own critic. In my experience, your best code comes from solid, objective reviews with constructive feedback. In a situation where you’re siloed in your abilities and have an entire department of an enterprise riding on your shoulders, justifying decisions and doing things in this way is not comfortable nor encouraging. Honestly, it’s stressful.

Given how large our industry is now, there’s no reason an open, nurturing place that anyone can post their code and get it reviewed doesn’t exist. Sure, you can say StackOverflow is there for small questions and you can get code reviews on StackExchange. If you’ve been in this industry for even a month, you know the culture of those platforms. The elitist, abrasive attitude that runs rampant in our industry shines on those platforms – everyone’s a 10x engineer there and everyone else is worthless in their eyes.

Sure, I could build this platform, but I have enough on my plate. In the process of builiding this platform, I stand in the same situation – no one to review my code unless I open source it, then who’s to say it’s going to gain any traction? Anyone that sees it has the reasonable justification that why waste their time on something that exists, albeit splintered across ten different platforms. The reason I say is this: we can only get the engineers we need by providing an environment that enables engineers to thrive without ridicule or animosity. Just because you understand some obscure computer science theory and think it’s common knowledge doesn’t mean everyone else does. Telling someone to RTFM or that their work is poor is not constructive.

I guess I’m just bitter that I personally am on an island. Isolated from other engineers, so my growth is stagnating. Isolated from upward or lateral mobility because I’m too important to the position I’m in. Isolated because my biweekly 1:1s don’t happen consistently. Sure, autonomy is great, but having the opportunity to speak to your direct manager when you’re having a difficult time justifying the work you do to yourself is not a good manager-employee relationship. Maybe my issue is I was so eager to break into an enterprise, I wasn’t prepared for the culture shock of it all and I’ve come to really dislike it. The candid interactions don’t exist, the rapid movements and agility to pivot doesn’t exist. The fifty stakeholders involved in almost every decision creates massive roadblocks at every turn.

Not to mention the performance metrics – those alone could be grounds to fire me. My counterparts in the department knocked out over 500 tickets last year – I think I did 150? Maybe 200? How will upper management look at that? They don’t understand the wide SQLi vulnerabilities I’ve patched nor the fact that I’ve consolidated thousands of lines of duplicate code into an MVC style framework. They don’t know what PSR is nor should they care. My time hits their bottom line and if I’m not as visually productive as my counterparts, why should I be in this seat?

They don’t understand that these things take time, especially when there’s no documentation that explains what the previous engineers were thinking, a decade ago. They don’t understand that a software engineer cannot do support, data entry, systems administration, database administration, and telephony administration on the flip of a switch. Each responsibility takes time to change mindset and each is a very different discipline. I was hired for a job, and the description I was given did not cover everything I now do. This is normal, to an extent. Gas lighting, however, is unacceptable and I’m not one ot be content with getting caught off guard by it.