Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 3835 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 3.4929. Min Range: -3.4929

Max Act: 3.4929. Min Act: -0.0000

Data Index: 1161383 (The Pile)

Max Activating Token Index: 136

Text #1

Max Range: 3.4929. Min Range: -3.4929

Max Act: 3.4348. Min Act: -0.0000

Data Index: 1593862 (The Pile)

Max Activating Token Index: 224

Text #2

Max Range: 3.4929. Min Range: -3.4929

Max Act: 2.0200. Min Act: -0.0000

Data Index: 599285 (The Pile)

Max Activating Token Index: 412

Text #3

Max Range: 3.4929. Min Range: -3.4929

Max Act: 2.2672. Min Act: -0.0000

Data Index: 1742885 (The Pile)

Max Activating Token Index: 202

Text #4

Max Range: 3.4929. Min Range: -3.4929

Max Act: 2.3242. Min Act: -0.0000

Data Index: 1651759 (The Pile)

Max Activating Token Index: 1016

Text #5

Max Range: 3.4929. Min Range: -3.4929

Max Act: 2.5402. Min Act: -0.0000

Data Index: 1758607 (The Pile)

Max Activating Token Index: 176

Text #6

Max Range: 3.4929. Min Range: -3.4929

Max Act: 2.2857. Min Act: -0.0000

Data Index: 1663555 (The Pile)

Max Activating Token Index: 597

Text #7

Max Range: 3.4929. Min Range: -3.4929

Max Act: 2.0590. Min Act: -0.0000

Data Index: 1647248 (The Pile)

Max Activating Token Index: 433

Text #8

Max Range: 3.4929. Min Range: -3.4929

Max Act: 2.2564. Min Act: -0.0000

Data Index: 1191378 (The Pile)

Max Activating Token Index: 261

Text #9

Max Range: 3.4929. Min Range: -3.4929

Max Act: 2.0761. Min Act: -0.0000

Data Index: 1179159 (The Pile)

Max Activating Token Index: 864

Text #10

Max Range: 3.4929. Min Range: -3.4929

Max Act: 1.9303. Min Act: -0.0000

Data Index: 674223 (The Pile)

Max Activating Token Index: 374

Text #11

Max Range: 3.4929. Min Range: -3.4929

Max Act: 2.0262. Min Act: -0.0000

Data Index: 830549 (The Pile)

Max Activating Token Index: 685

Text #12

Max Range: 3.4929. Min Range: -3.4929

Max Act: 1.9845. Min Act: -0.0000

Data Index: 1414751 (The Pile)

Max Activating Token Index: 551

Text #13

Max Range: 3.4929. Min Range: -3.4929

Max Act: 2.1074. Min Act: -0.0000

Data Index: 1755167 (The Pile)

Max Activating Token Index: 236

Text #14

Max Range: 3.4929. Min Range: -3.4929

Max Act: 1.9970. Min Act: -0.0000

Data Index: 1060328 (The Pile)

Max Activating Token Index: 753

Text #15

Max Range: 3.4929. Min Range: -3.4929

Max Act: 2.0988. Min Act: -0.0000

Data Index: 608304 (The Pile)

Max Activating Token Index: 638

Text #16

Max Range: 3.4929. Min Range: -3.4929

Max Act: 1.8599. Min Act: -0.0000

Data Index: 1838262 (The Pile)

Max Activating Token Index: 234

Text #17

Max Range: 3.4929. Min Range: -3.4929

Max Act: 1.9700. Min Act: -0.0000

Data Index: 1597830 (The Pile)

Max Activating Token Index: 370

Text #18

Max Range: 3.4929. Min Range: -3.4929

Max Act: 1.9346. Min Act: -0.0000

Data Index: 1361602 (The Pile)

Max Activating Token Index: 340

Text #19

Max Range: 3.4929. Min Range: -3.4929

Max Act: 1.8184. Min Act: -0.0000

Data Index: 1318142 (The Pile)

Max Activating Token Index: 513