My partner is in cybersecurity and I’ve been learning a lot from him about his field. My idea of a cybersecurity professional was someone pouring over logs looking for intruders and constantly trying to find ways to hack into their stuff to make it more secure. And while there is some of this (“SOC analysts” and “penetration testers”, respectively), the reality is more… corporate tbh.

It’s like when someone tells you they are a surgeon. You kind of know they use a scalpel and funny looking scissors-thing to sew people up but that is likely not 100% of their time and “well actually, we have people that close things up sometimes,” said a surgeon friend of mine. Anyway, “cybersecurity practitioners” is a very broad term, but it seems most are in the business of making sure they select tools that will be installed to catch when bad things happen. I learned, shockingly, that cyber teams are often more of an advisory role at organizations, and often not part of IT. They sit somewhere between IT/Legal/Advisor and many, like the rest of us, are frustrated their advice goes unheard and is not acted upon. But that’s a topic for another blog post, which I will not write because this isn’t a gossip column on things I know very little about.

Back to hacker summer camp, there are three conferences: Black Hat, where all the “corporate suits go to pitch their latest snake oil products” (hey man, I’m just quoting what I heard). BSides <location> is a fun concept: mini-conferences that are the “B Side” of the “A side” where the A-side is the main conference. (Cool kids into vinyl will understand the terms). For example, BSides San Franscisco is the B-Side to RSAC. They have them in just about every major city, worldwide, so if you want to get into cybersecurity check out of the city near you has a BSide conference. Anyway my partner gave a talk at BSides Las Vegas debunking the myth that 60% of medium and small businesses fail within six months of a breach. (Turns out the real number is more like ~35, for all time).

And then there is DEF CON (it’s in all caps, I’m not yelling at you, calm down). It’s hard to describe the audience because it seemed like everyone was welcome. Young and old, new or seasoned, nerdy or just crafty, it’s all there. It’s a little like a maker faire, but also comic con, but also a little citizen-hacky, but also ethical-hacker and a little crunchy.

DEF CON has a lot of parts. I saw someone post online “I have FOMO from missing DEF CON but more FOMO when I’m there because there’s so much to do and so little time.” This was very much true for me. In some ways it’s a very traditional conference schedule with talks, workshops, training, and demos. But there are also villages, communities, clubs, mini-cons, competitions, contests and parties. Best of all, no vendor expo and vendor booths. Instead the floor was sectioned into great areas where talks were delivered via the same headphones used for silent disco. Oh there were also so many stickers and beach balls and tinfoil hats. Did I mention the drum circle?

The villages are where each village organizer has brought you activities to do. There are villages on many topics like healthcare, aerospace, maritime, and they bring with them hardware, and data, and software, along with activities you can do. There’s a bomb diffusing village where every year someone makes a very complicated bomb-looking device that may take all weekend to successfully diffuse. My partner said one year he spent his entire DEF CON there and each team that stepped up to the plate learned from the teams before them. Even though there was only 2 people per team, by the end, it had become a community trying to beat the clock. Hundreds “died” but every one had fun. (That was a weird phrase to type)

There was an automotive village where you could hack a very brand new VW electric bus. You can connect to the CANBUS and try to get through WiFi or Bluetooth, just as long as you didn’t take apart the vehicle, it was fair game. There was an entire boat and semi truck and container at the maritime village. There were even hardware boxes for whatever controls the containers they remove from ships. Why? So you can experience it, find a vulnerability, look at how it works. The Boat was donated from the Auxiliary Coast Guard and you could see people inside plugging in and out of systems to find entry points or work on one of the CTFs (capture the flag).

CTFs are like a scavenger hunt with progressively more difficult things to find. My first CTF was one Elastic brought, using their tool called Kibana. (Btw that’s a great way to demo a tool!) Turns out I’m pretty good at that one because I’ve been looking at HPC system logs my whole career. I thought I would have to sharpen my BASH-fu but it turned out GUI tools have gotten pretty good at helping find info but also to sniff out sus logins and help you do forensics on a breach.

The first village we went to was the soldering village. They gave us cute little boards with LEDs and we went to town soldering tiny resistors and capacitors etc. Halfway in I got a little overconfident and soldered all 3 LEDs backwards but the villages have these extremely supportive helpers and I learned 2 new skills: 1) if you cut the legs of an LED, you can tell the positive / negative side because it has this little flat side of the light “bulb” circumference. 2) you can reheat excess solder and as soon as it flows, slam your hand with board against the table so the molten solder falls out. A moment of embarrassment turned into valuable new skills and increased confidence.

DEF CON was also useful for my own HPC world. There were three talks on data center security. As you may know I’m building a research compute data center currently, so these were very useful since many of the systems they explored/exploited are systems that support datacenter operations like chillers and HVAC.

Lastly there was Hacker Jeopardy. I can’t remember laughing this hard at a conference tbh. I also learned I suck at port math. (Hint: Port 23 is telnet btw)

I also met some amazing folks too. It obviously helps to walk into such a conference with someone that already has a lot of friends. But sincerely, the people were so welcoming and fun, open and generous, but most of all, inclusive of where ever you are in your life and career. I am looking forward to attending more.

If you are creative and curious, DEF CON is for you. And if you also choose to attend, keep it fun and weird… and DFIU. (You’re going to have to attend to get that insider joke).

The Vibe

The overall vibe at ISC is cozy in that the attendance is smaller than SC but in the best way possible. You get more interaction with other attendees and have the opportunity to have longer discussions, endless coke zero from fridges and coffee. This year I was deputy chair for posters, which in ISC terms is “chair in training” so I had some time to enjoy sessions before I’m fully in charge next year. I attended mostly Quantum sessions (of which there were 13!), and some sessions on user programs at other centers. I also attended plenaries and the Top500 session on which I have a lot of thoughts.

* The USA has left the chat *

In addition to the lower participation from US/DOE labs, the biggest change I noticed was that in almost all of the sessions I attended, the topic of European divestment of US tech was mentioned. Prepend the word sovereign to just about everything in HPC, like cloud, chip, hardware, software, quantum, etc, and it was said in many of the sessions I attended. EU governments seem determined to standup homegrown tech, and this has become a major priority the EU.

But, I didn’t get the impression that there was funding yet, so this effort appears to still be in the early stages. Ultimately, the rhetoric at ISC seems to be a mix of official government posturing and early groundwork for upcoming budget allocations. In a conversation with EU locals the general feeling is that the broader EU is highly motivated to make sovereign tech a reality but that the mechanism for EU funding has been fraught with, let’s say, “intergovernmental friction,” so that adds another layer of complexity. IMO, what will likely happen is subsidiaries of hyperscalers will simply offer local only options and cut the cables across the pond. But until that happens or more funding comes, it’ll likely take a decade.

(See the Dare-RISC Project that aims to develop 3 chip types.)

* China has entered the chat *

While Europe looks for a path to tech sovereignty, China showed up to brag that it built one. After nearly a decade of shunning/snubbing, nay ghosting, the Top500, China submitted a homegrown supercomputer that achieved #1, again. When Sunway Bluelight was launched 2011, much of the conversation then was about how far behind their homegrown CPU tech was. Chips in the west were already on 32nm tech, but Godson-3b used 64nm process, about two generations away. This time around, LineShine is a CPU-only machine based on an ARM core using a 7nm process. One can argue that is still 2-3 generations away but I think we’ve reached the point of diminishing returns on the power gains especially if you’re the type of country to brute force your way into increasing power capacity almost overnight. If power is ‘basically free’, does it matter if your system uses 30% more in power/flops?

Most of the conversation centered around how interesting the architecture was, and in particular, that it included no (NVIDIA) GPUs. (A few folks at the conference old enough to remember Larrabee felt some vindication too). Home grown interconnect, home grown software stack… Jack Dongarra’s closing keynote highlighted how Fugaku and LineShine are shining examples (pun intended) of industry and government engaging in co-design for their systems. The US scientific and HPC enterprise will continue to struggle on commodity hardware that is being primarily built to serve the needs of hyperscalers.

Shining light in the void

The Top500 felt flat this year. I could not be excited because I know these supercomputers are not the biggest and baddest anymore. Sure we could convince ourselves that HPC is tightly coupled storage+interconnect blah blah blah… But in a world filled with hyperscalers’ super-mega-giant-systems like Colossus, it all sounds like denial.

So much compute being deployed for chatbots, and not a drop for science to drink.

A nagging question lingers: why would China care to even submit again to the Top500? Why now, what’s the point, who was it for?

Dr. Lu Yutong’s talk I think had some clues. She covered the architecture showing completely homegrown software and hardware stack to an audience of European nations aspiring to do the same, at similar scale to LineShine. One might even say China has a solution to their problem, ready to go, for a European Union not averse to using tech from companies like Huawei. In the end, she invited the audience to collaborate, a departure from previous machines that were reserved for national defense or national research. Implied in the message was that this machine is an ambassador for open science collaboration.

What else… oh right, Quantum

I attended some of the 13 quantum related sessions, but one in particular stood out. It was about the quantum computing ecosystem. There was a panel on HPC + quantum computing integration, some representing industry, others places like BSC, Julich etc. Their CTA was, “join us in building the software.”

There are some 20+ quantum vendors, but only a handful of centers have HPC+QC on-prem. I am not sure how we can help build ecosystem when we don’t have access.

“The cloud” the vendors said.

“With a handful of free credits?” I asked.

Building an ecosystem with a trickle of free cloud credits with the understanding that you will pay soon or else it might be cut off, is not a great long term solution.

“You could sell us a tiny 3-4 qubit system for cheap so we can work on the operational aspect of the HPC+QC ecosystem” I suggested.

Well, with that comment I derailed that session, as others in the audience joined in agreement (sorry!). Vendors don’t quite understand what it’s like to integrate two systems together. There’s middleware, glue-code, security, multitenancy, scheduling, networking…

“We’ve been doing this for 20+ years, trust me, you just need the cloud.” a vendor said.

Anyway, the QC vendor that figures out how to give away their product to academics willing to do the free labor they need to build the ecosystem, will likely come out the winner. Build a collaboration so we can access your lab, give us a scratch-and-dent system, heck a hardware emulator would work if it has the same stack.

If you hold on to your tech so tightly that you’re afraid to have academics touch it, it’s going to be a long slog. Hardware is not useful without an ecosystem.

But that’s for another rant on another essay.

Other things I learned

• There’s a project that appears to be similar to the American Science Cloud called the Euro HPC Federation Platform (EuroHPC JU). (identity management is hard!)

  • Kenneth Hoste says “see my-eurohpc.eu to learn more”

• Datacenter build backlash doesn’t really seem to be happening in europe right now, at least not at the same scale as the US. The necessity for sovereignty is likely giving EU datacenter projects a ‘national security pass’ for now.

• Hartree has a computer named “Mary Coombs” where the name was chosen as part of a scicomm campaing for the launch around a human-centered story. Very effective.

• The Top500 has a new home in SigHPC

• Radlers continue to be delicious

Final thoughts

ISC26 was great, and I hope you’ll consider submitting a poster at ISC27! What are your suggestions for a good poster session?

Also, ISC had this neat AI generated summaries you can find here.

They also recently released a bunch of videos. Look at the HPC Around the world series to find out what’s happening globally! Very cool.

If you’re wondering who this internet rando is, I’m Fernanda Foertter. I work in supercomputing, HPC, AI, and scientific computing… I help people make computers do things with numbers.

You can find the official “yes, she’s real” page at fernandafoertter.com.

AI will be like fast fashion: it’ll cheapen software and create a lot of garbage, and we will still buy software from boutique shops.

An old colleague of mine said this today: “if software is basically free, what will that make us (software engineers)? What will be our role?”

First of all, that will never happen. Every token costs someone money, and right now it’s investors. In the future it’ll be enterprise, and it would be very unwise for enterprise to eliminate their engineers and be at the mercy of token providers. (sidetone: imagine being 85% of the way through a huge project and your token budget is over. How will you redefine the scope if they code-oracle can’t pump out more code?)

Model makers will eventually need to recoup investors funding and will raise the cost of tokens (or whatever the quanta for monetization of AI becomes); the same way cloud storage was cheap in the beginning, how did that work for us? Which reminds me…

AI is like cloud products circa 2010: an initial loss leader to get you hooked but wait till the real costs get passed on along with fees. So, many, fees….

Remember when cloud cost pennies and every one was like killing on-prem infra because they were led to believe you didn’t need all that staffing on the cloud? A decade later cloud costs ballooned, people re-built on-prem resources, and now they have cloud + on prem + cloud cost consultants. An entire industry was built around managing cloud costs. The same will happen with AI. I can only imagine that at some point AI bills will be riddled with fees in the same way that old phone bills had local city/county/state/federal taxes. It’ll be worth it because AI promises to cure disease overnight despite thousands of brains going at it for decades. Which reminds me…

AI reminds me of the Human Genome project: despite doomsday predictions, it’s taken 25 years for meaningful therapies related to gene editing to come to market.

If you can remember the days of the human genome project and the fear of everyone having designer babies and literal immortality, congratulations, you’re old enough for a colonoscopy. None of that came to pass, because actual life is harder than merely reading the “instructions for life". It turns out things get complicated quickly when you actually want to implement designer anything. I suspect the same will be for software-beyond-vibe-coded-demo. Take that demo and try to scale and things will break, fast. My first job was in agriculture, supporting a company doing genetically-assisted-breeding of livestock. You wouldn’t believe how hard it is to genetically engineer anything. Which reminds me…

AI reminds me of the rise of “industrial farming”: it feeds the world but we will still insist on locally sourced farm-to-table GMO-free organic free range pasture-raised hand-crafted software.

Enterprise might be bragging about tokens like they once bragged about how big their mainframes were. But in a decade I suspect they’ll brag about getting a seat at a boutique software company filled with software engineers who hung in there throughout the crazy AI days. Now, these engineers are the only ones left that actually know how to make things scale and connect and work. Once upon a time, when everyone had a database, consulting firms promising to optimize your DB queries popped up with cheap offshore labor. I’m sure they were brilliant optimizers but (in my experience) unless you understood the business pipeline, they did more harm than good. We paid someone a lot of money overseas to “optimize” our SQL queries an extra 10%, but never mind how the data layout and 300 table views accreted over two decades into a franken-store. Then someone put up a website promising a query optimizer as a “smart assistant” that took the burden of performance tuning away from developers. Sounds familiar, which reminds me…

AI reminds me of the Dot.Com boom: a domain land grab that made some very rich but also your aunt made a website to sell her weird art.

AI is a digital land rush all over again where fortunes are being minted overnight by the people who got there first, bought the right real estate, and convinced everyone else the future had already arrived. Back then it was domains, banner ads, and sock puppet mascots; now it’s GPU clusters, wrappers, and founders tweeting “agents” into existence before the product even works. To quote Mo Bitar “The bot doesn’t have to be intelligent, it just has to be more convincing than the median CEO is skeptical.”

But your aunt, launching a vibe-coded website to sell uncanny AI-generated fairy art and crystal astrology merch at 2 a.m., who is convinced she’s building the next Etsy empire, is the very democratization that fuels the fire. Like the early internet, the space is crowded with scams, experiments, genuine innovation, and deeply weird personal projects all coexisting in the same space-time continuum alongside some genuinely useful products. This noise makes it impossible to tell what’s revolutionary and what’s just glitter hot-glued to a landing page surrounded by dancing GIFs.

AI (LLMs) is Koko the gorilla using sign language, or those pets using speech buttons to tell their cat-dad it wants to go visit grandma. AI will not lead us to Planet of the Apes but listen: it’s amazing at taking notes during endless conference calls, transcribing videos, and as an assistive technology for people who are blind or like me, just who just want to pay attention to the person they are speaking with and not the notes.

For me this is revolutionary - and, enough. Which reminds me of this ComputerWorld magazine front page circa 2009. AI will come of Age… and actually I think it already has. We just need to accept that what it can do now is brilliant enough.

3. Simulation of Human Thinking

4. A Library for 2000 A.D.

5. The Computer in the University

6. Time-Sharing Computer Systems

7. A New Concept in Programming

8. What Computers Should Be Doing

I love looking back and how people imagined the future. It’s remarkable to me how we put so much trust in people predicting the future today when often when reading the past we see people really struggle to see beyond their own horizon. Someone in the 1800’s would have asked for a faster horse, it’s unlikely they would imagine an actual car.

The Library of 2000 A.D.

In Chapter 4, John Kemeny, the man who co-developed BASIC, imagines the future of libraries in the year 2000 where a “great central library” called the National Research Library, “automated with machine search.”

John would have been delighted to ask a reasoning LLM today to find everything there was to know about any subject he desired, with links.

Well, John, we were off by a quarter century but you got close enough. Gilbert W. King from IBM (he invented a method to translate Chinese) who was on the panel discussion corrects John, rather impolitely.

He goes on to predict PageRank as a form of “cross referencing” in search. None of them of course could have predicted this would be used for internet pages and not, you know, books.

Kemeny then replies to King with a burn.

One last comment from King I find interesting. If this is not an LLM description, I don't know what is. AI is compression after all.

Artificial Intelligence

The words “artificial intelligence” come up 26 times. They describe a future of something called the “General Problem Solver” or GPS for short. J. C. R. Licklider was a psychologist and computer scientist was an early pioneer in AI and ARPANET. He wrote a seminal paper titled “Man-computer symbiosis” where he wasn’t a fan of the idea where AI would take over. In that paper he wrote

In today’s parlance he says “ let the clankers do the boring stuff freeing us humans to do the important stuff, so ignore the people warning about AGI”. And he seemed spot on the level of compute needed to make this happen.

In contrast, M. Minsky was a pessimist worried about superintelligence warns “Future psychologists may envy the simple problems we faced when we had only to deal with human thought”

The idea that there will be a future profession of “AI Psychology” had not dawned on me but seems perhaps likely. There’s also a lot of fun ideas in this book about the future of computing, programming languages, parallel processing etc.

Computing as a Public Utility

Every once in a while you find people really talking about the car they just don’t know what it’s called yet.

I mean, the data phone is accurate, there just wasn’t a vocabulary then for fiber. In the end, it’s all waves moving along a wire, like a telephone.

The part where Mauchly seems to predict The Cloud™ is fascinating.

Ok but what does a utility do for people. For one utilities are indispensable to contemporary life. One can argue that humans can live without computers but I would argue the same could be said about electricity: we could all close up shop when the sun goes down and return to candle light and huddling by a fire. Food, warmth and shelter are basic necessities, but for modern life, I can’t imagine a world where electricity or computers are turned off completely. Compute resources are essential to modern life, and when they go down, (I’m looking at you AWS-US-East) well, chaos ensues.

Compute resources are certainly in the realm of infrastructure. It cannot be built at scale without significant capital. Think rail, water, sewage, power grid. Datacenters fit in the same realm. But the service of compute, that’s much more like a utility. Like a metro commuter service, the actual water through he pipes, the analog waves of a phone line and switch board, the actual generation of electricity and transmission… the services hyperscalers provide definitely feel like utilities.

I can’t imagine hospital care without computers, emergency services without computers, location/shipping services without computers, engineering design without computers. I can certainly imagine a world without these things, I grew up largely in that world, but I’m not sure I would like to go back to that world. And naturally something that requires a lot of capital limits the number of entities that can offer these services, which again feels a lot like a necessary utility. And up until a few years ago, the very large scale computers have been largely funded by governments, the last scale of which was the Exascale project. But only a few years after that we have seen unprecedented scaling by hyperscalers, well beyond the 35-40MW used to power what was then the world’s fastest supercomputer. This is an order of magnitude lower than current launched systems coming from hyperscalers.

Electricity can power a hospital or a stadium, and while both are part of living a full life, one is more essential than the other and we must make sure hospitals keep their lights on. Computation in the pursuit of science is essential in similar ways and we must carve out some of that compute infrastructure and funding to continue work in pursuit of better and safer outcomes through scientific methods and simulation.

(Skip to The Thread Tracker heading if you just want to know what I did)

During the holidays I decided to give vibe-coding an earnest try. So far I had only used ChatGPT for one project where I had to use SQL and it had been 15 years since I last touched SQL. I have ADHD so staying organized is the only way to keep the ADHD shut-down from happening.

Over the last two decades I’ve used many different organization systems and switched depending or the work environment I found myself in. When I was in school I used a lab book and a monthly calendar I would draw on my lab notebook. When I had my first job out of school I was mostly stamping out fires, so daily todo’s worked just fine. When I moved to ORNL however, I started to travel a lot and context switch a lot, so I turned to Bullet Journals (the OG not the ornate #BuJo stuff).

When I worked in Tech my job became primarily one as information sharer, so I took notes using an app called Workflowy. It worked great because it was bulleted and I could use “@” or “#” to organize my notes. It wasn’t perfect, things got buried in bullets and sub-bullets-of-sub-bullets . I loved Workflowy for how easy it was to take notes, it even had a menulet for MacOS and an iOS app. But while the bullets had depth, the tagging and topics had no way to describe contacts or entities. (Like who even is “@Bob” who does he work for, what is even his favorite cereal? I need to jot that down)

Another app that comes close to what I need is the Bear App. Bear is actually a lovely app that I highly recommend. I still can’t reference people or really take notes on them and it’s bound to a device with a $30/year subscription required if you want to sync across devices. So I can’t take notes on my Linux machine and later add to those notes when I’m lying awake at night thinking of that one time in 3rd grade I said something dumb and need to distract my brain by doing something else. I would be stuck to an iOS ecosystem and I’m a woman of many-OS layers.

There’s also Notion or Motion and I’m sure a myriad of others I could have been clever with how I organized my notes and gotten what I needed. But I don’t want to be clever and have to cajole some other app to do what I need. In a world of personalized algorithms, why can’t I personalize an app too?

So I put my dusty Product Manager hat on and asked what features I wanted out of an app:

• It needs to be fast entry. “Talk to Bob about topic regarding event.”

• I want to keep track of certain topics: “Talk to Bob about #topic regarding event”

• I want to keep track of people I come across so I can contextualize our conversations. “Talk to @Bob about #topic regarding event”

• I want it to help me keep track of events “Talk to @Bob about #topic regarding %event”

• I want it to keep track of entities: “Talk to @Bob, who works at $company, about #topic regarding %event”

• I want it to put on the calendar events and sub-events: “Talk to @Bob, who works at $company, about #topic regarding %event.deadline”

• I want it to organize priorities: “Talk to @Bob, who works at $company, about #topic regarding %event.deadline !!! ”

Should be easy right? Right.

The “Thread Tracker”

In my early Twitter days a decade ago, I realized that I could create a private account and use it for taking notes by leveraging hashtags for easy searching. It mostly worked to save tweets I wanted to reference later. I would RT and add my own hashtags, reply with my thoughts, add images, and had a TweetDeck open for the search functionality. It felt natural but it didn’t have all the things I wanted. Other apps I mentioned above came close but I need more.

I chose Lovable.dev for my vibe-coded app. Now I should preface: I am a developer. But I’m the kind of developer that can write applications to simulate materials and atoms and do chemistry on a supercomputer, using Vim (ok fine, I use Code, but I wanted you to get the picture). I’ve also dabbled in UI stuff like making a website for people to manage cluster jobs, I used to manage a BB forum way back in my college days. I was even part of a Gnome Summer of Code once… anyway, I’ve tried building UIs and I’m just not good at them. I know what I want, I don’t know how to do it and make it functional, and I lack the patience time to learn to do it right

But my first prompt Lovable hit it out of the park.

Prompt: I want to create a personal app to keep track of my tasks and treads. It works like this: there's single box where I enter a phrase with hashtags indicating a topic. to the right of the input box is place to insert a date for the future. these can be due dates, reminder dates, event dates etc. It's meant for rapid input. The app should keep a database of entries with 4 columns: date entered, the text, the hashtags found in the text, and the future date. beneath this box it should show the latest entries and they should be editable. I should also have a view that I can search all the text or hashtag, or date range. And if I click on a hashtag, a popup will appear with the latest entries related with that hashtag ordered from most recent to oldest.

Soon I had a calendar and ways to add deadlines. Woot!

And #todos that have a checkbox and when checked, they turn #todo’s into #completed (so satisfying)

When a priority hashtag is added an automatic #todo is added. So fast

Anyway we went back and forth like this for a while.

Add this feature.

No you dumb bot, you broke it again. Fix it.

Oh you’re right, I see the problem and now it’s fixed

Not it’s not. Ugh.

Lovable has this nice feature that you can go back to the prompt that broke things and it will roll the code back and try again. I found that if a prompt was too complicated it would sometimes break things and continue going down a bad path so I would just revert again and ask it to do something simple like change the color of something or:

Before I knew I had login, a database, documentation, registration, a tutorial… it was a, dare I say it, a functional app?

The Pithy App is Born

The Thread Tracker was fine as a name, but it wasn’t really an app name. So I renamed it to Pithy and was so excited I got a domain to go with it. You can check it out here—> www.pithyapp.com.

Caution! Don’t go posting you deep dark company secrets there, I can read the database and I haven’t figured out how to encrypt the text. The other stuff like user name and id are all hashes but the content itself appears to be plain text.

Speaking of security I have no idea if this has three backdoors, or other dumb things. My next step was to get Claude or GitHub AI to look at the GitHub repo and do an assessment. I unfortunately have zero web app security experience, so if you break it or break in, let me know! It’ll be a curious experiment.

At the very least I had a ton of fun playing with Lovable and I would love to build desktop app next. I would also love to learn if I can turn Pithy into a desktop app. Surely that should be possible with like Electron on something. I really don’t know what I’m doing but this has made me excited to want to learn more. Lastly I had not yet come across typescript in my line of work and looking at the codebase I’ve been learning more about it.

Conclusions

Anyway this was a fun experiment that I’ll consider a success for vibe-coding. For four days of “work” I could now take this prototype to a real developer shop and have a sharper idea of what I want out of a future app instead of giving people some sort of nebulous description of how a feature ought to behave.

Questions you might want to ask:

1. How much did this cost? I’d say it was less than $100 worth of credits on Lovable.dev… more like $50 probably, plus or minus $50.

2. Does it actually work? It does! it’s a bit wonky but so far I’ve actually been able to use it for a week and it seems to be working without any data loss. Sometimes I have to edit a note and reparse it and it gives me a bogus notification that doesn’t actually say that it picked up the new tag word I edited, but while the notification is wrong the action is correct. The tutorial also runs every time you login, not just the first time. Let’s call this a low priority bug for now.

3. Will the data search scale? I have no idea. What happens when I have hundreds of tags? Will it show me appropriate notes and not crash the browser? I don’t know.

4. Will it scale with users? Your guess is as good as mine. I didn’t intend to make this public, I was just playing around. I don’t expect this will even get read so….

5. What will it cost as you scale it? I don’t know, I didn’t intend to scale it at all. This is just a show-and-tell, calm down.

6. Is it secure? Oh I would not assume so. I just don’t know yet.

7. Will this technique work for computational chemistry apps? unlikely, there are just not enough examples. But you should try it out! Make sure to ask whatever AI you’re using for very specific features. That worked well for me.

If you’d like to learn more about what was implemented you can read below. This list was generated by Lovable’s AI and I’ve commented where it overstated a feature. I also added screenshots at the bottom. I hope you enjoyed my first Substack post and see you next time.

~~~~~***~~~~~

Features Implemented

Here’s a comprehensive list of everything Lovable and I built together:

Core Note-Taking System

1. Entry/Note Creation - Fast text-based note capture with real-time parsing

2. Rich Text Parsing - Automatic detection and linking of hashtags, mentions, entities, events, URLs, emails, and markdown links [text](url)

3. Due Dates - Schedule notes for future dates with calendar picker

4. Clear Due Date Button - Easy removal of due dates with an X button

5. Additional Details - Expandable long-form notes (up to 3,000 characters) attached to entries

6. Entry Editing - Full edit dialog with text, priority, dates, and details

7. Entry Deletion - Remove notes with confirmation (Author here: the confirmation is bogus)

Tagging System

8. #Hashtags - Categorize notes with topic tags

9. @Mentions - Reference people and auto-create person records

10. $Entities - Track companies/organizations with auto-creation

11. %Events - Link notes to projects/events with sub-events support

12. Hashtag Popup - View all entries for a hashtag with association management

13. Person Popup - View/edit person details (email, phone, role, context, linked entities)

14. Entity Popup - View/edit entity details (name, industry, location, website, linked people)

15. Event Popup - View/edit events with date/time, location, and linked people/entities

Priority & Task Management

16. Priority Shortcuts - ! → #optional, !! → #important, !!! → #critical

17. #todo / #completed Tags - Task tracking with checkbox toggles

18. Priority Filtering - Filter todos by critical/important/optional in the Todo view

19. Show/Hide Completed - Toggle visibility of completed tasks

20. Priority-Sorted Todos - Automatic sorting by priority level

Email Intelligence

21. Email Auto-Detection - Automatically extract email addresses from notes

22. Auto-Create Person & Entity - john@acme.com creates @john and $acme.com with linkage (Author here: this fails to create the connection between person and entity, but adding manually keeps the connection)

23. Email Autocomplete - Suggestions while typing email addresses

Follow-up System

24. %followup Parsing - %followup tomorrow auto-sets due date and adds #todo

25. Relative Date Parsing - Understands “tomorrow”, “next week”, “in 3 days”, “Monday”, “Jan 15”, etc. (Author here: yeah, this only works with #todo tag)

Calendar Integration

26. Calendar View - Visual calendar showing entries, events, and sub-events

27. Date Indicators - Colored dots showing what’s scheduled on each day

28. Selected Date Details - View all items for a selected date

29. Add Note for Date - Click calendar date to pre-fill due date in new entry

30. Upcoming Lists - Separate views for upcoming entries, events, and sub-events

Events System

31. Event Creation - Create events with %event-name

32. Event Details Page - Full page for editing events with all details

33. Sub-events/Milestones - Add deadlines and milestones within events

34. Event Location - Optional location/venue field for events (Author here: it’s just annotated, doesn’t do anything with location yet)

35. All-Day Events - Toggle for all-day vs timed events

36. Associated People & Entities - Link people and companies to events

37. Sticky Save Button - Save button stays visible while scrolling event page

Contacts Management

38. People Directory - View all people with search and filtering

39. Entity Directory - View all entities/companies with search

40. Activity Grids - Visual activity indicators (like GitHub contribution graphs)

41. Bi-directional Linking - People and entities auto-sync their associations

42. Orphan Cleanup - Delete people/entities no longer referenced in any notes

User Interface

43. 5-Tab Layout - Capture, Calendar, Todos, Events, Contacts

44. Dark/Light/System Theme - Theme switcher in settings

45. Responsive Design - Works on desktop and mobile

46. Beta Notice Banner - Alert for beta status

User Experience

47. Onboarding Tutorial - Step-by-step walkthrough for new users

48. Starter Notes - Optional creation of sample notes after tutorial

49. Re-run Tutorial - Option to replay tutorial from settings

50. Help Documentation - Comprehensive help dialog with examples

Data Management

51. Export to JSON - Download all notes as backup

52. Import from JSON - Restore notes from backup file

53. Reparse All Notes - Re-process notes with latest parsing logic

54. Statistics Dashboard - Count of notes, hashtags, people, and events

Authentication & Security

55. Email/Password Auth - Sign up and sign in

56. Password Reset - Forgot password with email link

57. Change Password - Update password from settings

58. Sign Out - Secure logout

59. Session Persistence - Stay logged in across sessions

60. Row-Level Security - Each user’s data is private. (Author here: define private? Admin can read all notes. I wouldn’t call that private)

Polish & Extras

61. PWA Support - Installable as a progressive web app

62. Ko-fi Support Link - “Buy me a coffee” integration

63. Contact Email - Direct email link in header

64. Markdown Links - Support for [display text](url) syntax

65. URL Auto-Linking - Clickable links in notes

Screenshots

At basic training, I would fall out of every run. But CSM Wills, a stocky older guy with biceps bigger than my legs and a low gravely voice likely damaged from yelling at thousands of young soldiers before me, would always drop back and finish the run with me. He was incredibly gentle, yet firm. “C’mon private. Look at me, I’m an old man. Don’t let this old man outrun ya.” I can still hear his voice in my head. If I stopped to catch my breath, he’d run slowly backwards until I started running again. When we got back to the barracks he’d just go on his way. No lectures, no shaming, no pep talk, just an unspoken understanding that I would try harder next run.

A few weeks in my DS called me upfront and told me to be squad leader for that morning’s run, standing right next to him. That position was the pace setter for our runs, everyone was literally looking to me to keep the formation steady. I didn’t fall out of that run or any run after that. Counterintuitively, being the pace setter meant people depended on me, not wanting to let people down, I found the energy to keep going. Most importantly, I didn’t die like my brain told me I would all the other times.

Back at my unit in Florida, being in the band also meant showing up for our communities during hurricanes, handing out food and water, and then playing concerts to lift spirits. Despite what we see on TV, military service isn’t all about soldiering or war. Sometimes it’s about meeting people where they are and giving them what they need in that moment, as a medic, a rescuer at sea, or even as a bugler (even one that isn’t very good at running). That lesson, finding strength when others are counting on you, stayed with me.

CSM Wills, I don’t now where you are now, but on this Veterans Day, I’m thankful you kept me running when I wanted to stop nearly 30 years ago. And to everyone who’s ever shown up for me since then, in their own way, to keep me running even when all I want to do is fall out and catch my breath. 🫡

(This article was originally published on LinkedIn)

That’s how it feels building a new supercomputing center from the ground up. One day you’re designing user programs and hiring world-class talent, the next you’re choosing carpet colors and deciding which coffee machine will keep your team at peak performance (because let’s face it, great science runs on caffeine). Feels very much like a startup and I’m thankful that experience can be applied here.

I drive by the site every day, watching walls go up and imagining what’s to come; the hum of compute nodes, the buzz of students learning HPC for the first time, and the spark of new collaborations waiting to happen, imagining the outcomes and the science this center will deliver. My mind is constantly swirling with ideas, programs, and initiatives that will help us not only join the global HPC community but also grow a vibrant local one.

It’s also been a growth opportunity for me personally. I’ve gone from living in jeans and hoodies to realizing that maybe, just maybe, it’s time to invest in a few boardroom-appropriate outfits. Building a center means showing up for everyone involved, from students to trustees, and learning to bridge those worlds with equal enthusiasm (and maybe slightly better shoes). This has been perhaps my hardest transition, still, I’m a work in progress.

However, this journey isn’t a solo endeavor. It’s a community effort powered by the support of university leadership, state partners, federal programs, and industry collaborators. It’s equal parts thrilling and humbling to build something that will train a new generation of students and empower researchers across so many fields. One bite at a time, and one very well-caffeinated team, we’re getting there.

We may not be the size of the exascale systems that makes the national headlines, but we are exactly the kind of center that science depends on, one that helps researchers model rivers, study materials, simulate storms, and train the next generation to push boundaries in their own way.

Anyway, windows starting going in today and I was very excited and decided to write up a little note. Eleven months to go until opening day... one bite at a time.

This article was originally posted on LinkedIn

Let’s recap some highlights over the years:
- In 2009, I watched with excitement as Jensen announced the GT300
- That same year, Leonardo 2.0 painted the Mona Lisa
- In 2013 I presented at GTC about the OLCF’s Titan supercomputer
- In 2017 I presented at GTC about OLCF’s upcoming Summit supercomputer
- In 2024 I attended as an employee of a startup building tools that run on GPUs!

Baby Fernanda meets Jensen's leather jacket for the first time.

I could not have imagined then that I would one day end up working for the company I admired so much, let alone that I would build two of the world’s fastest supercomputers with NVIDIA’s hardware. 2009-era Fernanda would have laughed at the thought.

Fast-forward to 2024, and I was as excited to attend as 2009. Perhaps more so, after a 3 year in-person hiatus, this year I attended as a proud employee of a GPU-based software startup!
So here are my top observations.


Overall impression of the event
One of the reasons GTC has been my favorite conference is that if you peeked into any room, you’d see code up on the screens. In the early days of CUDA, attendees were developers presenting their algorithms and implementations. The audience, also developers, would happily share their ideas, and what they had tried. NVIDIA devtechs would share a ton of very technical content. It’s not that this type of content wasn’t present, it’s just that it wasn’t as prevalent and I missed that kind of interaction. This year, the presentations from NVIDIA were largely about marketing their tools. To be fair, NVIDIA announced a lot of tools and there was only so much time. But things must evolve and GTC has certainly evolved since the early scrappy GPU developer community days.

Unlike AI, the early days of CUDA were more on the academic side without the clear external pressure to monetize. I don’t expect to see a return of those early GTC years, because if someone built something neat with AI, they will likely present as a startup, not as a developer. Regardless, there was still excitement in the air, and new toys to play with.

Here are a few talks I enjoyed:

Mixed emotions about mixed precision
If you want to infuriate an old “greyHPCer”, suggest to them that FP64 is not always necessary and that their work can be partly accomplished with lower precision. NVIDIA’s DevTechs have done some fantastic work to show this is indeed the case. Stephen Jones' talk on the future of CUDA was fantastic. He starts out reminding us that native FP64 precision = energy costs, because floating point arithmetic scales N^2 of the size of the mantissa. But you can use a fraction of the power using tensor cores and still get equivalent precision using mixed precision. He proved that if you can use FP16 TCMMA instead of FP64 FMA you can gain a factor of 20x in power efficiency.

While there is a lot of grumbling about the ever decreasing real estate of FP64 on the latest GPUs, the reality is that mixed precision is amazing and every HPC application developer should test and adopt mixed precision. NVIDIA developers have already provided performant libraries (cuBLASDx and cuBLASLt to name a few) that can help with that transition. Will it work for all HPC problems? No. But it seems likely that most HPC applications could benefit from adopting some level of mixed precision.

And here's the slide where they show reduced energy use.

Composability and Productivity with Legate
Before I talk about Legate, I want to say that the number of NVIDIA libraries have exploded and are getting difficult to keep track of. Many of the underlying numerical libraries are packaged under multiple frameworks, and framework names often don’t describe their function. It may be time for NVIDIA to step back for a bit and put together a comprehensive product map.

For example, it seems like the Legate/Legion path is non-existent on NVIDIA’s HPC webpage, yet it’s promoted as an HPC programming model. It’s part of the nvHPC ecosystem yet I couldn’t even navigate to anything Legate-related from the main HPC page, and it took far too long to figure out what it is and what it isn’t. In short, I think Legate is a NumPy replacement with an Apache Arrow data structure PGAS-like model with cuNumeric as a compute interface (instead of like, BLAS). (This is after going all over the place on the website.)

Anyway, what I love about Legate is its composability. If you’re new to this buzzword, you can think of composability as separating all workload into three main layers: interface, compute, and locality. In simpler terms, it’s: how I express what I want, a runtime that configures what’s needed, and the hardware environment it’ll run on.

If you think composability sounds a lot like DSL -> jit compiler -> optimized binary -> hardware, you have the right idea, but it’s even more generalized than that. Your interface could be some scientific application, ML, or just some post-processing. From there, an intermediate representation is generated and shipped to some runtime. That runtime could focus on scaling, or focus on running on accelerated hardware, or both. The runtime might be smart enough to understand data locality, and ship work next to the data, or the runtime may be smart enough to ship work to the most appropriate hardware based on some criteria.

Legate seems to imply you can go from local laptop to supercomputer without much effort. Legate also seems to imply it’s targeting productivity and scaling, not necessarily performance. Some might wonder why they might bother running on a GPU or supercomputer if not aiming for performance? The answer is capability. There’s probably more than enough performance to beat CPU-only performance and there’s a ton of “performance” you can reclaim from enabling productivity alone.

So much tech, so little time… The conference was over just when it was getting good. There was so much more announced at GTC which I’ve yet to really dive into, like FP4/FP6 and new algorithms using them. Another tool I learned about was NVIDIAs AI Workbench which I’ve installed and have started exploring. I might write something about my experience with this tool at a later time.

The future of HPC at GTC
In terms of HPC, GTC was more of a reunion even though many GPU developers chose to attend virtually. Some of my friends were discouraged by the levels of AI hype, and had concerns the event was going to be overwhelmingly AI-focused (they weren’t wrong). Another friend who attended in person suggested GTC be split into scientific computing and AI. Separating HPC GPU developers from AI practitioners would probably bring back some of the value for these folks that chose to attend virtually.

From where I sit in my career, smack in the middle of AI, data science and HPC, having AI and scientific computing collocated is still valuable for me, but I would like to see more academic talks next year. There was a ton of opportunity to have more technical talks on mixed precision, DPUs, the ever growing HPC python ecosystem… For example, major HPC center talks were largely about what the center does. There was too little focus on specific scientific outcomes and what they are accomplishing with the latest nvGPUs. NVIDIA is still pushing novel hardware and software, but the HPC content from major centers didn’t reflect that. By not hosting these academic talks at GTC, NVIDIA risks fragmenting the community into their respective domain specific conferences and we will lose GTC as an event where we can share common methods.

I do expect that next year will be more interesting in terms of AI outcomes. I don’t see AI winter coming just yet. The pace of AI methods has not slowed down and with everything being so new, I think most companies will still be barely scratching the surface of what is possible come next GTC. HPC is also ramping up their own AI-for-science efforts like digital twins, <domain>-informed AI, and explainable AI. Separating scientific computing from AI communities would be a mistake. AI benefits from the scientific rigor that computational scientists bring to the table and we will need the new methods they develop to help bring trust and fidelity to AI. I’m looking forward to seeing what people build over this coming year.

I generally intend to use two workflows in my daily life

• Paper based bullet journal (the simple original type)

• iPad for general notes
  

What actually happened, is that I used random sheets of paper / notes app / stickies / google tasks / iPhone Siri reminders... to "keep track" of what I needed to be doing. As you can see, fragmenting the system isn't really helpful in completing tasks. Same with note taking, I noticed I was either using a (separate) paper notebook, an iPad (which is often out of battery or too bright, or beeping at me too many notifications) and even 3x5 cards. Unsurprisingly I was losing track of my life because in general, if it's out of sight of the ADHD brain, it's out of mind. As I travel I might not have the right notebook to take notes with, anyway I think you get the point.

I discovered Bullet Journals in 2014 while working at ORNL. Working at a national lab can mean that you're involved in many separate activities at different levels with different entities. There are internal activities, inter-DOE activities, broader Federal activities, vendor related activities, community activities, conference activities... and most of these tracks had very little overlap. On the other hand, for note taking, I was using Evernote back then, and well, that's sort of died. The bullet journal method saved me and I was a loyal practitioner of it, until covid hit. I don't know what happened, but ever since covid all of my systems to keep track of my life broke down and what replaced them wasn’t good enough to handle times when things got a lot busier.

I was skeptical the that reMarkable tablet would help, but the person I started dating uses one and well, we tend to see things from new rose-colored-heart glasses whenever we fall in love 😍. My argument against yet-another-new-gadget is that I already have a note taking app on my iPad. But iPads are connected to the internet and well, "I could research <topic>" (opens new tab, and then another ten tabs) and now I'm off track again. "But look! I can do a webclip right onto my notes! your expensive e-ink thing isn't event connected to the interwebz" I'd say.

The boyfriend also has ADHD and calmly nodded while responding "that's the point."

In the before times, before wifi littered the friendly skies, I could only achieve intense hyperfocus inside a long airplane flight. Something about the confined space + very loud engine noise + what-else-is-there-to-do-ness of a flight was great for catching up on deep thinking. One time, on a flight to Europe, I managed to read two whole audiobooks (at 2x speed) on software development. That one flight led to creating the structure of the ORNL hackathons, IMO the best thing I think I've accomplished in my career thus far.

I resisted changing the system that had worked so well for me in the past. But it wasn’t working for me now. I figured I would just get the reMarkable and if I didn't like it I would return it. What happened is that this one device united bullet journaling and note taking into one and brought me back.

For example, you can create notebooks, so I created one just for my bullet journal, another for research notes, another for book notes, another for home projects. The beauty is you can insert pages / move pages / move text etc just like an iPad, so it's great for organizing notes especially if you want to go back to something and keep writing. In a paper notebook the pages would disconnect if I jumped into something else, like say conference talks. Now to go back to the notes I can insert pages and continue my learning. There are folders for organization and quick notes (kind of like a random sheet of paper)... but then... that's really all it can do. It's not a super complicated device. You can import a PDF, read it, highlight it with your stylus pen, write on top of the PDF pages, switch to another notebook, write some more, and then, that's it.

You read, you write. That's it.

And the battery lasts forever so the whole "I have a great idea but iPad battery is dead, where's the charger, no the long charger so I can write while it recharges, where's the outlet?" For the ADHD brain, it's all about momentum and if the momentum is disrupted, the moment(um) is gone.

With this new role I've taken up as Director of HPC, I'm having to do some research in preparation for 2024 and I've easily spent 2-4 hours a night in very deep focus. Even though I have access to the internet on the laptop, I'm able to stay on task because the act of taking notes is in itself taking my eyes off the browser screen to write. With this thing, the momentum is sustained and I find myself having to consciously step away from the task or else it'll be 2am and I'll mess up my very carefully nurtured sleep schedule. This hyperfocused"flow state" is something I actually love about my ADHD brain. And if something about my todo list pops up I can flip to another notebook add it and return to my task. Again, something about the confined space and features of the device helps me focus.

I do wish it had some features like copy/pasting images into notebooks using the desktop app, it would make it easier to annotate PDFs or graphs etc. But I have the iPad for that if need be. And I really wish that it had handwriting-recognition based search. My favorite app on the iPad, Notability, has this and it's awesome. I can also import my notebooks into Notability if I really feel I need this; and I might as I'm considering have started writing a book (over the holidays thanks to reMarkable!).

Anyway I hope this helps someone out there. Boyfriend was right and gets all the credit, so if it works for you too, you can thank him for this post.

• Foundational Models

• Data Friction

And neither of these are traditional HPC buzzwords. So what's happening?

Everyone is going foundational

Let's tackle the first and get that one out of the way. I remember a time when scientists, who were mostly focused on deterministic simulations, would have scoffed at the idea that AI could ever be useful in their particular domains. But this year was different: just about every major scientific domain was embracing AI experimentation, but some of these specifically cited foundational models.

One thing I love to ask at conferences is "what exciting thing are you working on lately" and let me tell you, there were a lot of people talking about building foundational models. For materials science, for biology, for healthcare and, the most surprising one, for climate science. Climate scientists are notorious for asking for things like "bitwise reproducibility" across hardware, libraries, and even across levels of compiler optimization. And not for nothing, their field is being constantly scrutinized at a world stage by not only peers but also members of the general public. So it's important for policy to be able to compare models from year to year.

I was happy to see this move to experiment to what is seemingly a "blackbox" in the context of deterministic simulations. And if there's any field out there with the absolute perfect amount of data to "do AI right" its the climate community. All of the conversations I had were still in the work-in-progress phase except one poster. I expect to see a lot more next year especially as the largest/fastest machines that can handle large climate datasets, Frontier and Aurora, (not NVIDIA based) have yet to build a similarly functional AI ecosystem for their respective hardware.

Data Friction

The other new-ish buzzword taking flight this year was data friction. Defining this is rather ambiguous but I think the most succinct way is to say "data is hard." So so hard. There were several workshops and BoFs on the matter. Data workflows, pipelines, computational steering... I mean, HPC has struggled with data-ing for a long time. This isn't new, for a long time we had a tiny-data-in-> simulate -> big-data-out model in HPC.

But in 2023 the folks in interesting scientific spaces are excited about a computing model that reverses that order:
big-data-in -> magic -> small-data-out. (The magic in the middle can be anything: simulation, analytics, ML.) And yes, we had to analyze big-data-out outputs in the past. But that's not the same as "out of all these extremely-detailed digital twins, give me 5 candidates that exhibit XYZ." And then keep iterating until the digital twins match real life. In my head I call it "weighted parameter permutation hell."

(I once again put a shout out to statisticians, the real MVPs who can help us make sense from the ocean of data we find ourselves in. Make Statisticians HPC Again)

A few events stood out that week

• A National Science Data Fabric to Democratize Data Access and Reusability (NSDF)

• Launching the National AI Research Resource (NAIRR) through a Coordinated Pilot (NAIRR)

• The 18th Workshop on Workflows in Support of Large-Scale Science (WORKS23)

I want to focus on NSDF and NAIRR. NSDF session appeared mostly university faculty led. The goal is to sort of map and connect data sets from lots and lots of sources. The second, NAIRR, is a bit more unclear. The NAIRR BoF had representatives from DOE, NOAA, NASA and NIH. There was a report that was written outlining this pilot and the panel started off outlining what each agency would bring to the table. The responses were what you might expect.

• DOE: “we will bring computers”

• The rest: “we offer data”

I stood up at that point and made “more of a comment than a question” type of remark. I told the panel that my entire career has been chasing data that is never offered in a computable format. When I worked at NVIDIA customers told me they wanted to do AI but their data was locked up in proprietary or domain-specific formats inside instruments. When I was a consultant at BioTeam the Pharma companies told me their data were locked up in 23 different databases and none of it is harmonized. Now while working at Voltron Data my DevRel team of engineers tried to use NIH and data from census and those are either locked up in pdf, dat, csv or gzip downloads over ftp. It seems to me that if NAIRR, NSDF and Globus could combine forces we could solve a lot of this because "silos gonna silo."

It's exahausting to come up with a great idea that combines say weather data plus census data and fail because data is so hard to wrangle. That's data friction. And that's why I'm at Voltron Data. I'm tired of wrangling. I just want to compute, I want to filter, I want to see what's inside and not depend on some vague file description or incomplete metadata to know if it's worth downloading 7.5TB's of uncompressed csv via slow ftp.

Well this retrospective took a turn to ranting about the state of data, so I better close with a positive. Over 14,000 people attended SC23. WOW. What an exciting time to be alive.

What interesting buzzwords did you hear at SC23?

Today some "tech luminaries" asked for a moratorium on large AI models. Kind of like that time that Szilard and a bunch of scientists asked the US Government not to use the atomic bombs they just helped to build. Three weeks later two bombs were dropped on Japan.

First of all, ignore the stupid longtermism-based [4] paper above. If you want to understand what kind of world we are about to enter with AI, I suggest learning some nuclear history. I understand if you feel this is hyperbolic, but the parallels are uncanny in that the same nuclear technology that could save us from climate change is also the same that could kill millions when repackaged onto an ICBM. The irresponsible way we went about testing nuclear technology is the reason we collectively fear anything nuclear. In its early history, nuclear science was carried out so nonchalantly that we still suffer the devastating effects from the times we went about willy-nilly blowing up islands, deserts, ships and living things. [5], [6] And then there are the nuclear accidents that, despite government oversight, were handled so irresponsibly that it forever marred relationship with have with such a promising electrifying technology.

AI House of Mirrors

If you fear AI becoming sentient, don't. AI is nothing but a mirror. It'll reflect back what is put in front of it, like a parrot. I cannot be creative anymore than a mirror can make me look more buff when I am in front of it. Creativity requires experience, and artistic expression often comes to us from experience with tightened emotions. AI may look like it can create but the analogy of that creating is carnival mirrors. It can alter/detect things that are similar to what it has seen. Without experience or emotions it can only riff on what's put in front of it.

But that's also part of the problem. Put a bunch of "criminal faces" in front of AI to train and the answer to what an American criminal looks like will likely be a black male. Without good ethical humans giving it context, AI will reflect the worst of what humanity has already done. The biggest AI model we could ever build will not ever have the random mixture of neurotransmitters, emotional experiences, and external stimuli that human brains have. Therefore AI cannot imagine a better and more just future for us if we don't show what that future should look like.

Irresponsible AI will cause damage. It may be temporary but the length of that timeline is unknown right now. A decade? century? How long until we realize we need trained artists/architects/musicians in our lives again? One of the worst longterm outcomes for the future of AI is a population losing trust. AI could manage power grids for efficiency, AI could simulate floods and spare loss of life, AI could give us better answers on how to fix our climate. But fear could destroy this research and cause more unnecessary suffering along the way.

I think we will find that Bad-AI is a lot like a nuclear bomb: it needs humans to proliferate, and grow more damaging. Good-AI, like nuclear energy, has to potential to really change quality of life for the entire world. But people in power only see the world in zero-sum, and for someone to win using AI, they'll contend, there will need to be losers too.

AI ins't inherently bad, but AI needs applied ethics more than ever. It's time us nerds stopped asking if something *could* be built and ask an ethicist if it *should* be built before we start building. It's hard not to feel a bit of anger regarding this letter, after all, tech giants have been eliminating AI ethics departments everywhere. Without government to put on some breaks on the very real possibility of AI being used to hurt humans, the ones that engineers could ask whether the work they were about to embark would have consequences, are having their jobs eliminated.

We should ask for a moratorium on AI: a moratorium on the use of AI without ethicists. A moratorium for development of AI for use in weapons. A moratorium for AI for use on policing. There are clear places to pause AI until we understand training data and bias better. AI is too good a tool to be shut down and too powerful to use without rails. AI should not cause more suffering today than our society already causes.

If you'd like to understand more about ethical AI space here are a couple of sources. I may add more to this space in the future.

AI has a Burnout Problem - Great article showing the pressure teams and companies are under to deliver AI

DAIR - Timnit Gebru and her team

Sources

[1] "Pause Giant AI Experiments" https://futureoflife.org/open-letter/pause-giant-ai-experiments/

[2]. Szilard Petition https://ahf.nuclearmuseum.org/ahf/key-documents/szilard-petition/

[3] The Dangerous Ideas of “Longtermism” and “Existential Risk https://www.currentaffairs.org/2021/07/the-dangerous-ideas-of-longtermism-and-existential-risk

[4] Against longtermism https://aeon.co/essays/why-longtermism-is-the-worlds-most-dangerous-secular-credo

[5]Bikini Atoll Nuclear Testing https://en.wikipedia.org/wiki/Nuclear_testing_at_Bikini_Atoll

[6] Victims of Nuclear Testing https://www.nti.org/atomic-pulse/downwind-of-trinity-remembering-the-first-victims-of-the-atomic-bomb/

[7] https://www.technologyreview.com/2020/12/04/1013294/google-ai-ethics-research-paper-forced-out-timnit-gebru/

[8] Documentation of datasets and trained models https://cacm.acm.org/magazines/2021/12/256932-datasheets-for-datasets/abstract

[9] AI has a burnout problem https://www.technologyreview.com/2022/10/28/1062332/responsible-ai-has-a-burnout-problem/

I recently found notes from my first Supercomputing conference in 2009. First line read "RDBMS is incapable of scaling" and it reminded me of a nice project I worked on in 2010.

Fresh out of Gradschool, I landed at this agricultural company called Genus Plc. I was the "HPC Manager and Scientific Programmer" there, but after parallelizing a few of their production applications the first few months, I realized they also had a (big) data problem. I was about to add Data Engineer to my already long title, before it was cool.

I worked out of one of their subsidiaries, PIC, meaning the Pig Improvement Company. PIC was headquartered near Nashville, TN, and used phenotypic and genotype data to breed seedstock. They weren't trying to create super pigs, at least not in the sense the media portrays franken-livestock. The goal was to create animals that were more robust and healthier. Simply breeding your "best pigs" the old way can lead to inbreeding and undesirable results.

The first application of this approach was used to breed out a gene that caused animals to heat up and die from heart attack. The industry developed a panel to identify this faulty ryanodine receptor gene and started to remove these carriers from the herd (i.e. not breed them). Before this gene was known, the way to test for this condition, know as porcine stress syndrome, was for a vet to give a young pig anesthesia through the snout and listen for an elevated heart rate. If left undiscovered, the animals would die from any stress, even just being transported for example.

PIC collected decades of phenotypic data on their animals and they were accumulating an ever increasing amount of genomic data. The cost of genetic testing had dropped considerably and they went from some 200+ genes on a panel to a whopping 60,000. Deep sequencing data was also starting to come in from vendors like Illumina and BGI. When it was just a few SNPs the SQL database handled the data ok. So as to have as little impact on the business users, a data dump was done more or less weekly, at nights and weekends. But things were getting slower and at 90GBs for a single animal, each data drop was harder and harder to search through.

My knowledge of databases at that time was limited to running a (now defunct) website that died because I botched a database migration to a new host. But PIC was spending a lot of money on consultants to optimize our database, and with naiveté and Dunning-Kruger at my side, I set out to solve this problem. I proposed we move out of the big database and move into a new one just for the research team during a meeting and I'll never forget the words of geneticist in charge: "You computer people are all the same, you make big promises and leave us with complicated broken systems." oof, tough crowd.

What they meant was that if it's not something they can understand and maintain, they rather not be dead in the water after I move on. And this absolutely had come from past experience. Valuable lesson learned, and I pivoted my approach by asking my manager if I could sit in the meetings with the database consultants. I needed to understand the data more, but more importantly I wanted to understand why a relational database was buckling under the weight of what seemed at the time, a reasonable amount of data for it to handle.

But it's late and to continue with what happened next will require a second post. See you soon and thanks for reading!

Happy Computing Y'all.

The Problem with HPC Kids Today

First, let us clearly define the problem:

• New hardware come and go

• Applications outlive hardware

• Programming models can’t keep up

You’ll have to pardon the doodles but I felt like drawing this week.

Below is the lifetime of a typical HPC app on a time axis. Sorta. This is probably a community application, developed by several people, sometimes written in mixed languages, with several forked versions, all doing their best to squeeze out performance of the system they are currently using to accomplish their science.

Sometimes a developer doesn’t want to depend on someone else’s library. So they’ll go rogue, build their own libraries, often to the detriment of performance (don’t do it, I beg thee). As my colleague and friend Michael Wolfe (PGI) used to say “PhDs have been written just to squeeze performance out of this particular algorithm.” And when those algorithms didn’t exist he would say the corollary “PhDs are waiting to be written on this topic.”

If Type A apps are full of dependencies, the other extreme is Type C apps with zero dependencies. Somewhere in the middle is Type B apps. More on this in a bit.

So Type A app, say a large community supported application, with 20+ years of development history and scars along the way, is probably filled with design choices no one really knows why it’s that way. “Legacy” they’ll say. Probably that choice was based on the system the original developer had access to.

The GPU Hackathon Series events I created uncovered many of these. Once was a 17yo application for particle track tracing used a Brookhaven National Lab. First two days a group of five domain science developers who were not the original developers, tried to make sense of the data structures. Whenever they pulled one pointer or rearranged a struct to better fit the memory layout, it would break. Finally at the end of day 2 (of 5) I asked

“do you understand the physics enough to write your own app?”

“of course” they said confidently, they are particle physicists after all

“then why not start from scratch?”

A day later they had a much shorter (LoC), efficient, and infinetly faster application, done, written in C++ using Kokkos. Their vision for what could be done was limited on what had been done. Even brilliant scientists, it seems, are vulnerable to a psychological heuristic known as anchoring. It was so prevalent that I began to read books on psychology and group dynamics. I became a master manipulator (for good), able to redirect even the most stubborn of “but that’s not how it *should* work” people.

I came across many such examples during my tenure as training lead at ORNL. The CFD application that had an absurd number of nested loops with peculiar loop bounds. Probably some legacy L2 caching optimization that sat there, undocumented. The master c++ developer, that spent a whole day walking through debugger output, declaring all compilers he tried trash since it couldn’t figure out how to optimize that important, yet slow, kernel he had written. With each of these personas, I would first attempt logic, then negotiate, finally manipulate (for good) to get them unstuck from that sort of unproductive loop thinking.

The HPC community as whole carries a burdensome anchor. What started as a field of hero developers able to create and port an application on just about any system they had access to, has morphed to large community applications with a number of developers, strugling to keep up with hardware changes and requiring additional human effort to help. (See DOE early science programs such as NESAP, CAAR, and ESP)

“Friendly users” and not “user friendly”

While I was too young to have been there in the early days of HPC, through the magic of the time machine that is Google, we can get a glimpse of what it looked like back-in-da-day…

The article, by Michael Hannah of Sandia National Labs, describes how Cray was starting to get pressure to compete with new commodity hardware called a PeeCees (ok fine, IBM PC). Here are more excerpts

“As new generations of supercomputers supplant the existing machines, they’ll require completely new software to run them. I expect this trend for speed at the expense of user interface to continue… There’s a great need for the large and the small, but the in-between computer is neglected.” — Michael Hannah, SNL

Preach brother Michael, preach.

But here is the example of the anchor I mentioned earlier:

“To take advantage of the Cray-1’s speed scientists must modify their thought processes and program code to conform to its hardware architecture. In contrast, the PC’s advantage is that its interfaces are easily modified to conform to the needs of both users and other computers”

Did you catch that? In the very early days of HPC, way back when we were all young an innocent (and I was like, 7yo) the psychology that would anchor the future of HPC was laid out: if you want to use the big rigs, you gotta pay the price and do most of it yourself. It’s the price of innovation. That may have been true in the early days of one-off systems, but today’s systems are largely built using commodity hardware, why should users bear the early brunt of making things work?

As a hobby I enjoy reading about the olden days of supercomputing, especially when I’m travelling. In the many articles I’ve read the story is pretty much the same and Michael is not the first nor the last to point this expectation out. Add every workshop I’ve ever been to and the question is “who will build the tools?” and “whose responsibility is it to maintain xyz?”

Mythical Creatures of HPC

If everything is up to the user of the HPC system they will need a very broad set of skills to make their applications work. We know them as Hero Programmers and in the early days it was necessary to be skilled at everything because computing was the primary limiting factor, so there was only so much one could really compute.

It was both simpler and more complicated back then. Equations were truncated approximations, the number of particles limited to merely megabytes of memory. The science was limited by what one could do on a computer and the difficult work was how to creatively squeeze out more science (read math) than the system could do if coded naively.

Today’s systems are another matter, with processors, co-processors, specialty cores, multiple memory levels/sizes/bandwidths/latencies, intrinsics, and communication between CPUs, GPUs, I/O, even fabric based offload collectives. Today’s applications are multi-physics, multi-modality, multi-modular, multi-language, multi-system, multi-developer…

Oof.

With the help of my daughters I developed a new set of HPC personas for today’s modern HPC professional.

The hero programmer is now known as The Cerberus. They are gatekeepers of HPC, and have high expectations that you should exhibit all three types of expertise that they claim they possess, or else “you don’t belong here.” Fortunately these are rare.

The most common is the Phoenix, aka computational scientist, living somewhere between domain and computer sciences.

Griffins are powerful majestic creatures, able to squeeze performance out of any hardware. The library eggs they build for us are pure gold.

Then there are Dragons. These are rare, but responsible for generating deluge of datasets. They can be found in projects such as SKA, LIGO, CERN. (Pardon my Physicist bias, let me know where else you’ve seen HPC Dragons flying).

Unlike workshops, the GPU Hackathons allowed me to play with partnering these different personas with developer teams of large applications. What I observed was that combining computer scientists, with enough knowledge of the underlying hardware, together with domain scientists, with enough knowledge of the tools being used to access performance, worked best. These teams were productive, collaborative and yielded the best results especially when, individually, not a single person on the team was a cerberus.

New Lines Drawn

“This is fun and weird and all, but what about Type B applications you mentioned above?”

Glad you asked. Type B apps are what all HPC apps aspire to be. They are big (LoC), performant, feature rich, have a large user community of users, and can run on just about any system. Type B apps cannot wait, because they have a large number of users that rely on them. If the institution users depend on for compute buys new hardware, the app needs to run, well, and keep science moving. Therefore Type B apps cannot cater to every architecture, they will never have enough developers to tweak kernels, and they can’t wait on community based libraries to be developed if they want to stay at the bleeding edge of technology.

My opinion based on my experience (don’t @ me), is that for Type B apps to thrive they’ll need to delineate duties within a perf x feature ownership/leadership graph shown below (PeFOwL? help).

On the community side, domain scientists keep the responsibility of app maintenance and feature development, while computer scientists own major aspects of data movement, the greatest bottleneck affecting HPC today.

On the vendor side, domain scientists they hire are responsible to develop libraries of fundamental algorithms used in their respective domains, while computer scientist they hire are responsible to develop performant math+other libraries for their hardware.

If I had a wand, every vendor would claim ownership of the layer of performance libraries for every piece of hardware they make available. I would even concede that their libraries could remain closed source IF they agree with the communities they serve on a similar interface and make sure it’s extra performant where possible.

All of this is a long winded way to say: we need to renegotiate with vendors where our responsibility ends and their’s begins, and we must raise the anchor dropped in the early calm days of HPC if we are to chart a new course in this stormy sea of very sophisticated heterogeneous hardware we find ourselves in.

(Thank you for indulging me on the Harry Potter + sailing references)

Let me take you back to 2008 ish. OpenMp v3.0 had just been released in May. DOE supercomputers were dominated by multicore cpus, bluegene and cell architecture.

They are all very different but at their core (ha) is the concept that once you reach a compute unit, you thread ~4/8/16 way. So looked a lot like scaling discrete cpu nodes to the developer. You had to tweak it based on FPUs or cache but that high level was pretty much the same.

With low thread counts OpenMP 3.0 is perfect. You can tweak the number threads to say, lower some memory access pressure or ALU use. Jaguar for example had share FPUs so to get performance on some apps you had to cut in half number of threads.

Then Oak Ridge plans an upgrade to Jaguar using hot new things called GPGPU. 2010 compiler like PGI, Cray, PathScale, CAPS all start playing around with “what does it mean to thread across hundreds or thousands of cores. They prototype pragmas as part of this exploration. Two years away from Titans launch and the field is still grappling with pragmas as an alternative to CUDA. PGI develops CUDA Fortran and OLCF’s early science program is underway. OpenMP ARB is having intense arguments about where to take the standard next.

At the heart of the argument are strong egos who shall remain nameless. OpenMP wanted to be conservative, and justifiably didn’t want to react too quickly to what could possibly be a one-off arch in DOE (Titan). User feedback back then was basically “WTF DOE Why do you hate us?!” OpenMP wasn’t budging and ORNL was getting nervous because egos are blocking a needed exploration. So ORNL asks Cray (vendor), NVIDIA, AMD, PGI, CAPS to coalesce their individual efforts into a single standard w/ the hopes that something usable would exist while OMP ARB warmed up.

So a year before Titan’s launch, Nov 2011 OpenACC is born with the subset of OpenMP members who believed the future lay in 1000s of threads on accelerators. Boy were they right... (shout out to Aurora project for finally seeing the light.)

The memeber went to work combining their individua accelerator directives protypes into one open standard. It was modeled after OpenMP by design: OpenMP is easy to learn, plays nice with legacy apps, had clear syntax, and there was the intention that one day the two would merge.

OpenMP leadership at the time never accepted the split, even though these members continued to serve in OpenMP. It was infantile to put it mildly. ORNL now has an alternative to CUDA. S3D fork becomes first application to be ported to a premature OpenACC 1.0.

I joined ORNL in March 2012 and began to serve on OpenACC committee. OpenMP finally launched an accelerator committee and I began to serve there too. By this time the philosophy of these two standards couldn’t have been more divergent.

OpenMP 3.x assumes the application will basically find its way to a compute unit (think MPI to a node, or cell processor) and when a loop region appears it fans out a bunch of threads to accomplish the task. Data movement? non loop regions that aren’t tasks?

Titans GPUs needed data movement first. And the concept of kernels, aka chunks of work that would get shipped to the GPU. Need atomic? Tough luck, once the threads launched they were unreachable until complete. Take the longest loop and put it up top, move tiny loops inside.

But something else happened. The first ORNL Hackathon was sponsored by OpenACC. They wanted to see their users implement it. We noticed OpenACC wasn’t something you sprinkled on your code and turned it on/off with a flag. Everyone had to change the code.

We realized that the basic promise of a directive placed in comments that accelerate an application wasn’t what was happening because legacy apps were developed for low core high cache architectures. But when they changed it to work with OpenACC, apps ran better.

Enter a short lived architecture that rhymes with Tights Manding. An accelerator somewhere between bluegene and gpu, with wide vectors. Finally OpenMP is ready to indulge the accelerator fangirls, except the it basically took OpenMP 3.x, added data movement and SIMD wide vectors

Cray refocused on that architecture (for understandable reasons) It wasn’t until 2018, a full six years after Titan was launched that the first OpenMP 4.5 compiler was working thanks to Cori... and finally an OMP 4.5 based application ran on Titan.

Conclusion: So when you see the OMP/ACC “debate” what you’re really seeing is leftover heat of the Directives universe “Big Bang” aka people picking sides. But as user liaison for many years here’s my take: pick what works for you. Try them both. Try them all.

What I like about OpenMP is that its member base brings a lot of different architecture considerations. Short/wide vectors, discrete, embedded, big cache, are all there. At the same I don’t like is the legacy induced franken directives it’s become where nothing ever deprecates.

What I like about OpenACC is something I along with

@sunitachandra29 and others put in place: it’s user centric model. Users present at the first day of every F2F meeting. The committee prioritizes features based on requests, it doesn’t turn into a CS exercise in completeness.

We have usergroups on Slack, user group meetups at every major HPC conference. Hackathons. F2F meetings. User user user user... see a pattern? I tried this approach at OpenMP but frankly it didn’t take.

So at the heart of the philosophy it’s about a standard that is slower to adapt and able to have longer term perspective over architecture (OpenMP) vs one that jumps into new architecture and prioritizes based on current user needs. (OpenACC) will OpenACC turn into Frankenstein?

Who knows will they merge? Very very unlikely. Is OpenACC growing? Yes Did OpenMP get influenced ( and basically pushed into action) by OpenACC? Yes. Which one should you use? Try them both, try them all. Do what works for you. /fin #apologiesfortypos

I want to add one more thing. The current OpenMP board is filled with great pragmatic people.

This post was adapted from a twitter thread before I left the site.