Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 5229 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 1.3589. Min Range: -1.3589

Max Act: 1.3589. Min Act: -0.0000

Data Index: 1034971 (C4 (Web Text))

Max Activating Token Index: 544

Text #1

Max Range: 1.3589. Min Range: -1.3589

Max Act: 1.2780. Min Act: -0.0000

Data Index: 769322 (C4 (Web Text))

Max Activating Token Index: 833

Text #2

Max Range: 1.3589. Min Range: -1.3589

Max Act: 1.3195. Min Act: -0.0000

Data Index: 1210622 (C4 (Web Text))

Max Activating Token Index: 548

Text #3

Max Range: 1.3589. Min Range: -1.3589

Max Act: 1.3310. Min Act: -0.0000

Data Index: 968118 (C4 (Web Text))

Max Activating Token Index: 975

Text #4

Max Range: 1.3589. Min Range: -1.3589

Max Act: 1.1198. Min Act: -0.0000

Data Index: 382647 (C4 (Web Text))

Max Activating Token Index: 848

Text #5

Max Range: 1.3589. Min Range: -1.3589

Max Act: 1.0598. Min Act: -0.0000

Data Index: 1142923 (C4 (Web Text))

Max Activating Token Index: 729

Text #6

Max Range: 1.3589. Min Range: -1.3589

Max Act: 1.1111. Min Act: -0.0000

Data Index: 70131 (C4 (Web Text))

Max Activating Token Index: 96

Text #7

Max Range: 1.3589. Min Range: -1.3589

Max Act: 1.0947. Min Act: -0.0000

Data Index: 785087 (C4 (Web Text))

Max Activating Token Index: 592

Text #8

Max Range: 1.3589. Min Range: -1.3589

Max Act: 1.1159. Min Act: -0.0000

Data Index: 750930 (C4 (Web Text))

Max Activating Token Index: 87

Text #9

Max Range: 1.3589. Min Range: -1.3589

Max Act: 0.9808. Min Act: -0.0000

Data Index: 344115 (C4 (Web Text))

Max Activating Token Index: 825

Text #10

Max Range: 1.3589. Min Range: -1.3589

Max Act: 0.9972. Min Act: -0.0000

Data Index: 657428 (C4 (Web Text))

Max Activating Token Index: 1012

Text #11

Max Range: 1.3589. Min Range: -1.3589

Max Act: 0.9408. Min Act: -0.0000

Data Index: 39924 (C4 (Web Text))

Max Activating Token Index: 889

Text #12

Max Range: 1.3589. Min Range: -1.3589

Max Act: 1.0251. Min Act: -0.0000

Data Index: 1226836 (C4 (Web Text))

Max Activating Token Index: 412

Text #13

Max Range: 1.3589. Min Range: -1.3589

Max Act: 1.0252. Min Act: -0.0000

Data Index: 1353629 (C4 (Web Text))

Max Activating Token Index: 56

Text #14

Max Range: 1.3589. Min Range: -1.3589

Max Act: 0.9957. Min Act: -0.0000

Data Index: 1117147 (C4 (Web Text))

Max Activating Token Index: 300

Text #15

Max Range: 1.3589. Min Range: -1.3589

Max Act: 0.9013. Min Act: -0.0000

Data Index: 1617259 (Python Code)

Max Activating Token Index: 969

Text #16

Max Range: 1.3589. Min Range: -1.3589

Max Act: 0.9920. Min Act: -0.0000

Data Index: 792040 (C4 (Web Text))

Max Activating Token Index: 255

Text #17

Max Range: 1.3589. Min Range: -1.3589

Max Act: 1.0141. Min Act: -0.0000

Data Index: 416006 (C4 (Web Text))

Max Activating Token Index: 669

Text #18

Max Range: 1.3589. Min Range: -1.3589

Max Act: 1.0327. Min Act: -0.0000

Data Index: 293111 (C4 (Web Text))

Max Activating Token Index: 155

Text #19

Max Range: 1.3589. Min Range: -1.3589

Max Act: 0.7927. Min Act: -0.0000

Data Index: 815305 (C4 (Web Text))

Max Activating Token Index: 958