Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 5869 in Layer 11

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 1.3978. Min Range: -1.3978

Max Act: 1.3978. Min Act: -0.0000

Data Index: 722154 (C4 (Web Text))

Max Activating Token Index: 807

Text #1

Max Range: 1.3978. Min Range: -1.3978

Max Act: 1.0633. Min Act: -0.0000

Data Index: 1069019 (C4 (Web Text))

Max Activating Token Index: 767

Text #2

Max Range: 1.3978. Min Range: -1.3978

Max Act: 0.9303. Min Act: -0.0000

Data Index: 1286725 (C4 (Web Text))

Max Activating Token Index: 790

Text #3

Max Range: 1.3978. Min Range: -1.3978

Max Act: 0.8502. Min Act: -0.0000

Data Index: 803135 (C4 (Web Text))

Max Activating Token Index: 692

Text #4

Max Range: 1.3978. Min Range: -1.3978

Max Act: 0.7910. Min Act: -0.0000

Data Index: 640780 (C4 (Web Text))

Max Activating Token Index: 94

Text #5

Max Range: 1.3978. Min Range: -1.3978

Max Act: 0.6839. Min Act: -0.0000

Data Index: 483640 (C4 (Web Text))

Max Activating Token Index: 523

Text #6

Max Range: 1.3978. Min Range: -1.3978

Max Act: 0.6662. Min Act: -0.0000

Data Index: 1087733 (C4 (Web Text))

Max Activating Token Index: 128

Text #7

Max Range: 1.3978. Min Range: -1.3978

Max Act: 0.6788. Min Act: -0.0000

Data Index: 505324 (C4 (Web Text))

Max Activating Token Index: 608

Text #8

Max Range: 1.3978. Min Range: -1.3978

Max Act: 0.6109. Min Act: -0.0000

Data Index: 1044490 (C4 (Web Text))

Max Activating Token Index: 543

Text #9

Max Range: 1.3978. Min Range: -1.3978

Max Act: 0.6449. Min Act: -0.0000

Data Index: 1299445 (C4 (Web Text))

Max Activating Token Index: 167

Text #10

Max Range: 1.3978. Min Range: -1.3978

Max Act: 0.5855. Min Act: -0.0000

Data Index: 1357126 (C4 (Web Text))

Max Activating Token Index: 1022

Text #11

Max Range: 1.3978. Min Range: -1.3978

Max Act: 0.6239. Min Act: -0.0000

Data Index: 1523436 (Python Code)

Max Activating Token Index: 188

Text #12

Max Range: 1.3978. Min Range: -1.3978

Max Act: 0.6675. Min Act: -0.0000

Data Index: 138929 (C4 (Web Text))

Max Activating Token Index: 960

Text #13

Max Range: 1.3978. Min Range: -1.3978

Max Act: 0.6074. Min Act: -0.0000

Data Index: 664266 (C4 (Web Text))

Max Activating Token Index: 674

Text #14

Max Range: 1.3978. Min Range: -1.3978

Max Act: 0.5763. Min Act: -0.0000

Data Index: 433530 (C4 (Web Text))

Max Activating Token Index: 76

Text #15

Max Range: 1.3978. Min Range: -1.3978

Max Act: 0.5646. Min Act: -0.0000

Data Index: 1588403 (Python Code)

Max Activating Token Index: 154

Text #16

Max Range: 1.3978. Min Range: -1.3978

Max Act: 0.6032. Min Act: -0.0000

Data Index: 1182330 (C4 (Web Text))

Max Activating Token Index: 282

Text #17

Max Range: 1.3978. Min Range: -1.3978

Max Act: 0.5650. Min Act: -0.0000

Data Index: 194117 (C4 (Web Text))

Max Activating Token Index: 199

Text #18

Max Range: 1.3978. Min Range: -1.3978

Max Act: 0.5474. Min Act: -0.0000

Data Index: 221362 (C4 (Web Text))

Max Activating Token Index: 636

Text #19

Max Range: 1.3978. Min Range: -1.3978

Max Act: 0.4874. Min Act: -0.0000

Data Index: 688206 (C4 (Web Text))

Max Activating Token Index: 927