Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2011 in Layer 11

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 3.4300. Min Range: -3.4300

Max Act: 3.4300. Min Act: 0.0000

Data Index: 369246 (C4 (Web Text))

Max Activating Token Index: 147

Text #1

Max Range: 3.4300. Min Range: -3.4300

Max Act: 3.1822. Min Act: -0.0000

Data Index: 242311 (C4 (Web Text))

Max Activating Token Index: 595

Text #2

Max Range: 3.4300. Min Range: -3.4300

Max Act: 3.0387. Min Act: -0.0000

Data Index: 865501 (C4 (Web Text))

Max Activating Token Index: 997

Text #3

Max Range: 3.4300. Min Range: -3.4300

Max Act: 2.2798. Min Act: -0.0000

Data Index: 648112 (C4 (Web Text))

Max Activating Token Index: 45

Text #4

Max Range: 3.4300. Min Range: -3.4300

Max Act: 2.1934. Min Act: 0.0000

Data Index: 671332 (C4 (Web Text))

Max Activating Token Index: 546

Text #5

Max Range: 3.4300. Min Range: -3.4300

Max Act: 2.2082. Min Act: -0.0000

Data Index: 576218 (C4 (Web Text))

Max Activating Token Index: 513

Text #6

Max Range: 3.4300. Min Range: -3.4300

Max Act: 2.0476. Min Act: -0.0000

Data Index: 986036 (C4 (Web Text))

Max Activating Token Index: 980

Text #7

Max Range: 3.4300. Min Range: -3.4300

Max Act: 2.1081. Min Act: -0.0000

Data Index: 925201 (C4 (Web Text))

Max Activating Token Index: 900

Text #8

Max Range: 3.4300. Min Range: -3.4300

Max Act: 1.8222. Min Act: -0.0000

Data Index: 1528401 (Python Code)

Max Activating Token Index: 586

Text #9

Max Range: 3.4300. Min Range: -3.4300

Max Act: 1.6141. Min Act: -0.0000

Data Index: 1415109 (Python Code)

Max Activating Token Index: 803

Text #10

Max Range: 3.4300. Min Range: -3.4300

Max Act: 1.5124. Min Act: -0.0000

Data Index: 715788 (C4 (Web Text))

Max Activating Token Index: 35

Text #11

Max Range: 3.4300. Min Range: -3.4300

Max Act: 1.5472. Min Act: -0.0000

Data Index: 20062 (C4 (Web Text))

Max Activating Token Index: 369

Text #12

Max Range: 3.4300. Min Range: -3.4300

Max Act: 1.3057. Min Act: -0.0000

Data Index: 926995 (C4 (Web Text))

Max Activating Token Index: 503

Text #13

Max Range: 3.4300. Min Range: -3.4300

Max Act: 1.3668. Min Act: -0.0000

Data Index: 611216 (C4 (Web Text))

Max Activating Token Index: 307

Text #14

Max Range: 3.4300. Min Range: -3.4300

Max Act: 1.3588. Min Act: -0.0000

Data Index: 1523149 (Python Code)

Max Activating Token Index: 1015

Text #15

Max Range: 3.4300. Min Range: -3.4300

Max Act: 1.1529. Min Act: -0.0000

Data Index: 383218 (C4 (Web Text))

Max Activating Token Index: 383

Text #16

Max Range: 3.4300. Min Range: -3.4300

Max Act: 1.2895. Min Act: -0.0000

Data Index: 990777 (C4 (Web Text))

Max Activating Token Index: 667

Text #17

Max Range: 3.4300. Min Range: -3.4300

Max Act: 1.1951. Min Act: 0.0000

Data Index: 747258 (C4 (Web Text))

Max Activating Token Index: 580

Text #18

Max Range: 3.4300. Min Range: -3.4300

Max Act: 1.4259. Min Act: -0.0000

Data Index: 804667 (C4 (Web Text))

Max Activating Token Index: 722

Text #19

Max Range: 3.4300. Min Range: -3.4300

Max Act: 1.1159. Min Act: -0.0000

Data Index: 1414349 (Python Code)

Max Activating Token Index: 838