Loading…
This event has ended. View the official site or create your own event → Check it out
This event has ended. Create your own
View analytic
Sunday, February 9 • 09:50 - 10:30
fedmsg - Fedora Infrastructure's realtime message bus

Sign up or log in to save this to your schedule and see who's attending!

Description
-----------

In the Fedora Infrastructure team, we developed a realtime messaging layer on top of 0MQ to send and receive messages to and from our many and varied services. Debian Infrastructure has adopted it as well laying the basis for a multi-distro realtime message bus.

In this presentation, I'll give an overview of fedmsg -- the Federated Message Bus -- what it is, how it works, and why some design decisions were made the way they were. The second half will cover projects that integrate with fedmsg.

Abstract
--------

- What fedmsg is good for

- A way to keep your finger on the pulse of open source development
- A source of community metrics over time
- A platform for building out reactive realtime infrastructure

- What is Fedora Infrastructure? "How a Linux Distribution gets made"
- Overview of our topology -- what services are tied in
- Design decisions necessitated by an Open Infrastructure

- Signed, but unencrypted
- No central broker
- Public zmq.PUB socket

- CLI tools and API examples
- Applications built against fedmsg

- End user notifications for everyone - Desktop/Android/IRC
- Push to test/qa infrastructure
- Push notifications to mirrors
- Sync apprentice and packager credentials
- Visualizations - Faking git logs for "gource"
- Reports - fedora-news, this-week-in-fedora, & owner-change-tool
- Fedora Badges for distro contributors

- Debian adoption - Imagine a multi-distro realtime development pony

Outline
-------

- What fedmsg is good for

- A way to keep your finger on the pulse of open source development
- A source of community metrics over time
- A platform for building out reactive realtime infrastructure

- What is Fedora Infrastructure? "How a Linux Distribution gets made"

- What do packagers do?
- What does release engineering do?
- How do we communicate, make decisions, govern ourselves?
- Design and development.
- Testing, QA, bug reporting.
- All the good things in life.

- Overview of our topology -- what services are tied in

- [just talk about this diagram](http://threebean.org/presentations/fedmsg-flock13/images/fedmsg-flock13-img/topology.png)

- Design decisions necessitated by an Open Infrastructure

- We have an open infrastructure. You can show up at our weekly IRC meeting and get "fi-apprentice" rights to log in to machines.
- Signed, but unencrypted. Anyone can read, anyone can write. Only some messages are trusted. Key here is that anyone can debug at any point in our infrastructure.
- No central broker. Datacenters are donated and distributed. Things can go down. fedmsg designed to not bring *anything* down with it.
- Public zmq.PUB socket. The world can read at ``tcp://hub.fedoraproject.org:9940`` -- point a ``zmq.SUB`` socket at it with ``recv_multipart``.

- CLI tools and API examples

- CLI: ``echo "hello world" | fedmsg-logger``
- CLI: ``fedmsg-tail --really-pretty``
- API: ``fedmsg.publish(...)``
- API: ``fedmsg.tail_messages(...)``

- Applications built against fedmsg

- fedmsg-notifications. -- Problem: all of our applications carry their own email code. With that comes further baggage and maintenance.
With fedmsg notifications for interesting infrastructure events, we can put all that code in one place where it can be more easily maintained.
Benefit to the end-user: manage notification preferences in one place instead of per-app.
Opens up notifications to different contexts: Email? IRC privmsg? Android? RSS?

- Push to test/qa infrastructure.
When new packages are built, we can instantly run depchain analysis tests. This is was done previously with cronjobs.
When new composes are complete, we can fire off jobs in beaker to make sure install and integration tests pass. This was done previously by hand.

- Push notifications to mirrors
Over 200 mirrors right now. They have cronjobs that poll with rsync, we can use ``fedmsg-trigger`` to fire off rsync only when the compose notification is published.

- Sync apprentice and packager credentials
Ugly cronjobs exist that run that sync shell groups to machines from the Fedora Account System. With ``fedmsg-trigger``, we fire off an ansible playbook to sync creds when group-membership-change messages are published.
Same goes for package ownership and ACL changes. People had to wait up to an hour for their creds to propagate.

- Visualizations - Faking git logs for "gource".
Run "fedmsg-tail --gource | gource -i 0 --log-format custom -" to get the realtime version of [this](http://threebean.org/so-i-turned-the-fedmsg-data-into-a-git-log-and.webm).
Its the best office dashboard ever.

- Reports - fedora-news, this-week-in-fedora, owner-change-tool, & release engineering dashboard.
HTML5 dashboards that query the history of fedmsg to make useful reports.
http://ambre.pingoured.fr/fedora-news/
http://ambre.pingoured.fr/thisweekinfedora/
https://lists.fedoraproject.org/pipermail/infrastructure/2013-June/013070.html
https://apps.stg.fedoraproject.org/releng-dash/

- Fedora Badges for distro contributors
https://badges.fedoraproject.org
Backends listens to the bus and wakes up in response to new events.
Compares messages against a series of 100 or so rules defined in YAML. If a
contributor matches some criteria, then they are awarded a badge (achievement
unlocked!)
Fun fact -- this is hooked into Mozilla's Open Badges Infrastructure (OBI). https://openbadges.org/

- Debian adoption - Imagine a multi-distro realtime development pony

- This is amazing. Their bus is still coming up but they have a ton of message types already passing over it.
- They too have a publicly subscribable endpoint. Your end user machine can subscribe to *both* the Fedora and the Debian buses.
- Imagine this scenario: a security bug gets filed in Debian. A Fedora daemon listens to their bus and automatically files the same bug on our tracker against the same package (perhaps it is smart and checks for dupes first).
- A new package update is marked as a security fix in Fedora, a debian daemon listens and tries to check if the same patch is applied there. If not, it emails the maintainer to find out whats up.
- We just started talking about a Holy Grail - integrating compatible semantic DOAP and FOAF information with all messages so we can easily correlate activity between the distributions.
- Can we provide a service to upstreams? Aim your github service hook at some bucket service. When you tag a new release, it notifies the upstream-bucket, the upstream-bucket notifies anyone who wants to know (Fedora, Debian, & friends).

Speakers
avatar for Ralph Bean

Ralph Bean

Senior Software Engineer, Red Hat, Inc
Ralph Bean works as a Senior Software Engineer on the Fedora Engineering team at Red Hat. Most of what he does goes on in #fedora-apps on freenode: the application-development side of Fedora Infrastructure.


Sunday February 9, 2014 09:50 - 10:30
Lecture room D1

Attendees (17)