Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2320 in Layer 3

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.8068. Min Range: -1.8068

Max Act: 1.8068. Min Act: -0.0001

Data Index: 1309915 (C4 (Web Text))

Max Activating Token Index: 395

Text #1

Max Range: 1.8068. Min Range: -1.8068

Max Act: 1.6402. Min Act: -0.0001

Data Index: 295814 (C4 (Web Text))

Max Activating Token Index: 990

Text #2

Max Range: 1.8068. Min Range: -1.8068

Max Act: 1.3750. Min Act: -0.0001

Data Index: 1321484 (C4 (Web Text))

Max Activating Token Index: 875

Text #3

Max Range: 1.8068. Min Range: -1.8068

Max Act: 1.3121. Min Act: -0.0001

Data Index: 1042038 (C4 (Web Text))

Max Activating Token Index: 750

Text #4

Max Range: 1.8068. Min Range: -1.8068

Max Act: 1.3869. Min Act: -0.0001

Data Index: 1209261 (C4 (Web Text))

Max Activating Token Index: 395

Text #5

Max Range: 1.8068. Min Range: -1.8068

Max Act: 1.3910. Min Act: -0.0001

Data Index: 1200728 (C4 (Web Text))

Max Activating Token Index: 241

Text #6

Max Range: 1.8068. Min Range: -1.8068

Max Act: 1.1913. Min Act: -0.0001

Data Index: 221268 (C4 (Web Text))

Max Activating Token Index: 327

Text #7

Max Range: 1.8068. Min Range: -1.8068

Max Act: 1.2766. Min Act: -0.0001

Data Index: 1007155 (C4 (Web Text))

Max Activating Token Index: 27

Text #8

Max Range: 1.8068. Min Range: -1.8068

Max Act: 1.2487. Min Act: -0.0001

Data Index: 609055 (C4 (Web Text))

Max Activating Token Index: 541

Text #9

Max Range: 1.8068. Min Range: -1.8068

Max Act: 1.2310. Min Act: -0.0001

Data Index: 104260 (C4 (Web Text))

Max Activating Token Index: 300

Text #10

Max Range: 1.8068. Min Range: -1.8068

Max Act: 1.2637. Min Act: -0.0001

Data Index: 1209161 (C4 (Web Text))

Max Activating Token Index: 764

Text #11

Max Range: 1.8068. Min Range: -1.8068

Max Act: 1.1881. Min Act: -0.0001

Data Index: 587669 (C4 (Web Text))

Max Activating Token Index: 384

Text #12

Max Range: 1.8068. Min Range: -1.8068

Max Act: 1.1858. Min Act: -0.0001

Data Index: 50660 (C4 (Web Text))

Max Activating Token Index: 180

Text #13

Max Range: 1.8068. Min Range: -1.8068

Max Act: 1.1857. Min Act: -0.0001

Data Index: 646202 (C4 (Web Text))

Max Activating Token Index: 601

Text #14

Max Range: 1.8068. Min Range: -1.8068

Max Act: 1.2048. Min Act: -0.0001

Data Index: 1067913 (C4 (Web Text))

Max Activating Token Index: 582

Text #15

Max Range: 1.8068. Min Range: -1.8068

Max Act: 1.1326. Min Act: -0.0001

Data Index: 1312989 (C4 (Web Text))

Max Activating Token Index: 1007

Text #16

Max Range: 1.8068. Min Range: -1.8068

Max Act: 1.1891. Min Act: -0.0001

Data Index: 73095 (C4 (Web Text))

Max Activating Token Index: 390

Text #17

Max Range: 1.8068. Min Range: -1.8068

Max Act: 1.1805. Min Act: -0.0001

Data Index: 54804 (C4 (Web Text))

Max Activating Token Index: 460

Text #18

Max Range: 1.8068. Min Range: -1.8068

Max Act: 1.0730. Min Act: -0.0001

Data Index: 171251 (C4 (Web Text))

Max Activating Token Index: 187

Text #19

Max Range: 1.8068. Min Range: -1.8068

Max Act: 1.1529. Min Act: -0.0001

Data Index: 1184885 (C4 (Web Text))

Max Activating Token Index: 830