Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 5459 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 2.5109. Min Range: -2.5109

Max Act: 2.0973. Min Act: -0.0000

Data Index: 631645 (C4 (Web Text))

Max Activating Token Index: 400

Text #1

Max Range: 2.5109. Min Range: -2.5109

Max Act: 2.5109. Min Act: -0.0000

Data Index: 1279829 (C4 (Web Text))

Max Activating Token Index: 371

Text #2

Max Range: 2.5109. Min Range: -2.5109

Max Act: 1.4994. Min Act: -0.0000

Data Index: 1326019 (C4 (Web Text))

Max Activating Token Index: 686

Text #3

Max Range: 2.5109. Min Range: -2.5109

Max Act: 1.3179. Min Act: -0.0000

Data Index: 529049 (C4 (Web Text))

Max Activating Token Index: 199

Text #4

Max Range: 2.5109. Min Range: -2.5109

Max Act: 1.3758. Min Act: -0.0000

Data Index: 201113 (C4 (Web Text))

Max Activating Token Index: 593

Text #5

Max Range: 2.5109. Min Range: -2.5109

Max Act: 1.3087. Min Act: -0.0000

Data Index: 627444 (C4 (Web Text))

Max Activating Token Index: 722

Text #6

Max Range: 2.5109. Min Range: -2.5109

Max Act: 1.2535. Min Act: -0.0000

Data Index: 374289 (C4 (Web Text))

Max Activating Token Index: 51

Text #7

Max Range: 2.5109. Min Range: -2.5109

Max Act: 1.3318. Min Act: -0.0000

Data Index: 830835 (C4 (Web Text))

Max Activating Token Index: 136

Text #8

Max Range: 2.5109. Min Range: -2.5109

Max Act: 1.2922. Min Act: -0.0000

Data Index: 48084 (C4 (Web Text))

Max Activating Token Index: 348

Text #9

Max Range: 2.5109. Min Range: -2.5109

Max Act: 1.3266. Min Act: -0.0000

Data Index: 830478 (C4 (Web Text))

Max Activating Token Index: 717

Text #10

Max Range: 2.5109. Min Range: -2.5109

Max Act: 1.2113. Min Act: -0.0000

Data Index: 956894 (C4 (Web Text))

Max Activating Token Index: 105

Text #11

Max Range: 2.5109. Min Range: -2.5109

Max Act: 1.2719. Min Act: -0.0000

Data Index: 1301738 (C4 (Web Text))

Max Activating Token Index: 103

Text #12

Max Range: 2.5109. Min Range: -2.5109

Max Act: 1.2422. Min Act: -0.0000

Data Index: 730489 (C4 (Web Text))

Max Activating Token Index: 400

Text #13

Max Range: 2.5109. Min Range: -2.5109

Max Act: 1.0242. Min Act: -0.0000

Data Index: 731864 (C4 (Web Text))

Max Activating Token Index: 147

Text #14

Max Range: 2.5109. Min Range: -2.5109

Max Act: 1.2715. Min Act: -0.0000

Data Index: 80300 (C4 (Web Text))

Max Activating Token Index: 57

Text #15

Max Range: 2.5109. Min Range: -2.5109

Max Act: 1.2629. Min Act: -0.0000

Data Index: 1163532 (C4 (Web Text))

Max Activating Token Index: 54

Text #16

Max Range: 2.5109. Min Range: -2.5109

Max Act: 1.2560. Min Act: -0.0000

Data Index: 1134817 (C4 (Web Text))

Max Activating Token Index: 429

Text #17

Max Range: 2.5109. Min Range: -2.5109

Max Act: 1.2451. Min Act: -0.0000

Data Index: 854235 (C4 (Web Text))

Max Activating Token Index: 140

Text #18

Max Range: 2.5109. Min Range: -2.5109

Max Act: 1.3717. Min Act: -0.0000

Data Index: 739599 (C4 (Web Text))

Max Activating Token Index: 414

Text #19

Max Range: 2.5109. Min Range: -2.5109

Max Act: 1.1196. Min Act: -0.0000

Data Index: 1112159 (C4 (Web Text))

Max Activating Token Index: 303