HackerTube
HomeLongsLikedHistoryWatch laterCategoriesPlaylistsSettings

Python at Massive Scale - Stephen Simmons, Neil Slinger

PyData1 HN recommender · recommended in 1 month, 2018-2022
YouTube
10K views · 8 years ago · talk · dated

A PyData London 2018 talk on how JPMorgan scaled its Athena Python platform over ten years to about 35 million lines of code and thousands of developers. It covers infrastructure, tooling and culture for analytics and visualization at that scale.

Cited as the definitive public account of Python at massive scale, especially in discussions of monorepos and bank Python.

Python, Ruby & scripting languagesSoftware practice, open source & careers#jpmorgan#athena#monorepo#python#tradingsystems#developertooling

8 comments from Hacker News

@Barrin922022 · in “Modern Python Performance Considerations”

>Python shines beautifully as a (somewhat) cross platform scripting language Python is much more than just a scripting language. I remember attending this talk[1] a few years about JPMorgan's 35 million LOC Python codebase. Python is being used to built seriously large software nowadays and I don't think performance is ever a minor issue. It should always be in the top 3 for any general purpose language because it directly translates into development speed, time and money. [1] ▶ this video

CitesView on HN
@stevesimmons2022 · in “Problems once solved by a metaclass can be solved by __init_subclass__”

See [1] for a PyData London talk I did on "Python at Massive Scale", about JPMorgan's Athena trading and risk management system, used for FX, Commodities, Credit, Equities, etc trading. At that time, Athena had 4,500 Python developers doing 20,000 commits a week. Also see one recent HN discussion on "An Oral History of Bank Python" [2]. [1] ▶ this video [2] link

CitesView on HN
@stevesimmons2021 · in “An oral history of Bank Python”

I worked on Athena at JPMorgan for 8 years, and loved it. Seeing Python at the core of trading, risk and post-trade processing for Commodities, FX, Credit etc was such a great developer experience. By the time I left JPM, there were 4500 devs globally making 20k commits weekly into the Athena codebase. (I did a PyData presentation on this [1] for more details). The one downside was the delayed transition from Py2.7 to 3; I left just as that was getting underway. [1] ▶ this video The

CitesView on HN
@stevesimmons2021 · in “An oral history of Bank Python”

Nice stuff! For a view of Python at JPMorgan, I did a talk at PyData London "Python at Massive Scale" - ▶ this video

CitesView on HN
@stevesimmons2020 · in “The most remarkable legacy system I have seen”

As the comment by timsworkaccount says, this is definitely Athena at JPMorgan. For those interested, I gave a talk on Athena at PyData UK 2018 called "Python at Massive Scale". 4500 developers making 20k commits a week. Codebase with 35m LOC. The video is here: ▶ this video It covers Athena's origins, what it is used for, application architecture, infrastructure, dev tooling and culture.

RecommendsView on HN
@Barrin922019 · in “Python Is Eating the World”

>How stable is python to run a full trading / quantitative algorithm on? JP Morgan operates a ~30 million LOC Python platform for trading and analytics. (related talk: ▶ this video ) Yes, there are very, very large working python codebases in fields out there that demand correctness. I'm honestly getting tired of the static typing circlejerk that has entered the industry.

CitesView on HN
@mcintyre19942018 · in “Python is becoming the world’s most popular programming language”

If anyone else is looking: ▶ this video :) I'm definitely interested in this - I've seen a struggle to manage a far smaller codebase than that in Python and really interested to see what you're doing differently to make it work.

CitesView on HN
@stevesimmons2018 · in “Why Google Stores Billions of Lines of Code in a Single Repository (2016)”

My company has a 50m LOC Python codebase in a monorepo. It works really well, given the rate of change of thousands of developers globally. That is only possible because of the significant investment in devtools, testing and the deployment infrastructure. Here is "Python at Massive Scale", my talk about it at PyData London earlier this year: ▶ this video

CitesView on HN
James Powell: So you want to be a Python expert? | PyData Seattle 20171:54:11

James Powell: So you want to be a Python expert? | PyData Seattle 2017

PyData
508K views • 9 years ago
Keynote: Begin...End?57:05

Keynote: Begin...End?

David Beazley
9K views • 5 years ago
Raymond Hettinger, Keynote on Concurrency, PyBay 20171:13:53

Raymond Hettinger, Keynote on Concurrency, PyBay 2017

SF Python
161K views • 9 years ago
Scalability at YouTube38:43

Scalability at YouTube

Next Day Video
51K views • 14 years ago
Nathaniel J. Smith - Trio: Async concurrency for mere mortals - PyCon 201832:17

Nathaniel J. Smith - Trio: Async concurrency for mere mortals - PyCon 2018

PyCon 2018
30K views • 8 years ago
What nobody tells you about documentation30:52

What nobody tells you about documentation

PyCon AU
105K views • 9 years ago
Raymond Hettinger - Dataclasses:  The code generator to end all code generators - PyCon 201845:08

Raymond Hettinger - Dataclasses: The code generator to end all code generators - PyCon 2018

PyCon 2018
130K views • 8 years ago
Larry Hastings - Removing Python's GIL: The Gilectomy - PyCon 201632:07

Larry Hastings - Removing Python's GIL: The Gilectomy - PyCon 2016

PyCon 2016
49K views • 10 years ago
Armin Ronacher - A Python for Future Generations42:44

Armin Ronacher - A Python for Future Generations

EuroPython Conference
6K views • 9 years ago
Transforming Code into Beautiful, Idiomatic Python48:51

Transforming Code into Beautiful, Idiomatic Python

Next Day Video
1.1M views • 13 years ago
Modern Dictionaries by Raymond Hettinger1:07:41

Modern Dictionaries by Raymond Hettinger

SF Python
150K views • 9 years ago
RailsConf 2011: Aaron Patterson, "Double Dream Hands: So Intense!"47:34

RailsConf 2011: Aaron Patterson, "Double Dream Hands: So Intense!"

O'Reilly
26K views • 15 years ago
Understanding the Python GIL46:10

Understanding the Python GIL

David Beazley
127K views • 14 years ago
CppCon 2018:H. Wright “Large-Scale Changes at Google: Lessons Learned From 5 Yrs of Mass Migrations”1:00:16

CppCon 2018:H. Wright “Large-Scale Changes at Google: Lessons Learned From 5 Yrs of Mass Migrations”

CppCon
16K views • 8 years ago
Christophe Pettus - PostgreSQL Proficiency for Python People - PyCon 20163:00:06

Christophe Pettus - PostgreSQL Proficiency for Python People - PyCon 2016

PyCon 2016
8K views • 10 years ago
Q&A with Guido van Rossum, Inventor of Python56:55

Q&A with Guido van Rossum, Inventor of Python

Microsoft Reactor
28K views • 5 years ago
Introduction to Julia - Part 1 | SciPy 2014 | David Sanders1:01:04

Introduction to Julia - Part 1 | SciPy 2014 | David Sanders

Enthought
80K views • 12 years ago
Google Developers Day US - Python Design Patterns44:03

Google Developers Day US - Python Design Patterns

GoogleDeveloperDay
90K views • 19 years ago
The Mental Game of Python - Raymond Hettinger [synced version]1:10:00

The Mental Game of Python - Raymond Hettinger [synced version]

Tech Ma
24K views • 6 years ago
Erlang Factory 2014 - That's 'Billion' with a 'B': Scaling to the Next Level at WhatsApp49:15

Erlang Factory 2014 - That's 'Billion' with a 'B': Scaling to the Next Level at WhatsApp

Erlang Solutions
50K views • 12 years ago