Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4542 in Layer 1

Load this data into an Interactive Neuroscope

See Documentation here

Hooked Transformer Loading: HookedTransformer.from_pretrained('solu-12l')

Click the toggle to see the full text!

Text #0

Max Range: 2.4036. Min Range: -2.4036

Max Act: 2.4036. Min Act: -0.0002

Data Index: 660525 (C4 (Web Text))

Max Activating Token Index: 105

Text #1

Max Range: 2.4036. Min Range: -2.4036

Max Act: 2.2658. Min Act: -0.0002

Data Index: 1226012 (C4 (Web Text))

Max Activating Token Index: 104

Text #2

Max Range: 2.4036. Min Range: -2.4036

Max Act: 2.1241. Min Act: -0.0002

Data Index: 116215 (C4 (Web Text))

Max Activating Token Index: 563

Text #3

Max Range: 2.4036. Min Range: -2.4036

Max Act: 2.1124. Min Act: -0.0002

Data Index: 357863 (C4 (Web Text))

Max Activating Token Index: 39

Text #4

Max Range: 2.4036. Min Range: -2.4036

Max Act: 2.0666. Min Act: -0.0002

Data Index: 1110149 (C4 (Web Text))

Max Activating Token Index: 145

Text #5

Max Range: 2.4036. Min Range: -2.4036

Max Act: 2.0570. Min Act: -0.0002

Data Index: 553554 (C4 (Web Text))

Max Activating Token Index: 400

Text #6

Max Range: 2.4036. Min Range: -2.4036

Max Act: 2.0265. Min Act: -0.0002

Data Index: 225204 (C4 (Web Text))

Max Activating Token Index: 500

Text #7

Max Range: 2.4036. Min Range: -2.4036

Max Act: 2.0443. Min Act: -0.0002

Data Index: 10596 (C4 (Web Text))

Max Activating Token Index: 205

Text #8

Max Range: 2.4036. Min Range: -2.4036

Max Act: 2.0356. Min Act: -0.0002

Data Index: 72745 (C4 (Web Text))

Max Activating Token Index: 313

Text #9

Max Range: 2.4036. Min Range: -2.4036

Max Act: 1.9958. Min Act: -0.0002

Data Index: 1247953 (C4 (Web Text))

Max Activating Token Index: 453

Text #10

Max Range: 2.4036. Min Range: -2.4036

Max Act: 1.9914. Min Act: -0.0002

Data Index: 595660 (C4 (Web Text))

Max Activating Token Index: 444

Text #11

Max Range: 2.4036. Min Range: -2.4036

Max Act: 2.0008. Min Act: -0.0002

Data Index: 575029 (C4 (Web Text))

Max Activating Token Index: 765

Text #12

Max Range: 2.4036. Min Range: -2.4036

Max Act: 2.0224. Min Act: -0.0002

Data Index: 340782 (C4 (Web Text))

Max Activating Token Index: 151

Text #13

Max Range: 2.4036. Min Range: -2.4036

Max Act: 2.0265. Min Act: -0.0002

Data Index: 964578 (C4 (Web Text))

Max Activating Token Index: 294

Text #14

Max Range: 2.4036. Min Range: -2.4036

Max Act: 1.8730. Min Act: -0.0002

Data Index: 553951 (C4 (Web Text))

Max Activating Token Index: 744

Text #15

Max Range: 2.4036. Min Range: -2.4036

Max Act: 1.9292. Min Act: -0.0002

Data Index: 264097 (C4 (Web Text))

Max Activating Token Index: 240

Text #16

Max Range: 2.4036. Min Range: -2.4036

Max Act: 1.9413. Min Act: -0.0002

Data Index: 491549 (C4 (Web Text))

Max Activating Token Index: 146

Text #17

Max Range: 2.4036. Min Range: -2.4036

Max Act: 1.9178. Min Act: -0.0002

Data Index: 1270983 (C4 (Web Text))

Max Activating Token Index: 206

Text #18

Max Range: 2.4036. Min Range: -2.4036

Max Act: 1.8359. Min Act: -0.0002

Data Index: 1083672 (C4 (Web Text))

Max Activating Token Index: 844

Text #19

Max Range: 2.4036. Min Range: -2.4036

Max Act: 1.9423. Min Act: -0.0002

Data Index: 908375 (C4 (Web Text))

Max Activating Token Index: 543