Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3441 in Layer 4

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.2328. Min Range: -1.2328

Max Act: 1.1976. Min Act: -0.0001

Data Index: 1287135 (C4 (Web Text))

Max Activating Token Index: 982

Text #1

Max Range: 1.2328. Min Range: -1.2328

Max Act: 1.1110. Min Act: -0.0001

Data Index: 413227 (C4 (Web Text))

Max Activating Token Index: 784

Text #2

Max Range: 1.2328. Min Range: -1.2328

Max Act: 1.1677. Min Act: -0.0001

Data Index: 699141 (C4 (Web Text))

Max Activating Token Index: 715

Text #3

Max Range: 1.2328. Min Range: -1.2328

Max Act: 1.2328. Min Act: -0.0001

Data Index: 1255118 (C4 (Web Text))

Max Activating Token Index: 643

Text #4

Max Range: 1.2328. Min Range: -1.2328

Max Act: 1.1873. Min Act: -0.0001

Data Index: 412831 (C4 (Web Text))

Max Activating Token Index: 184

Text #5

Max Range: 1.2328. Min Range: -1.2328

Max Act: 1.0916. Min Act: -0.0001

Data Index: 1267051 (C4 (Web Text))

Max Activating Token Index: 535

Text #6

Max Range: 1.2328. Min Range: -1.2328

Max Act: 1.1099. Min Act: -0.0001

Data Index: 364664 (C4 (Web Text))

Max Activating Token Index: 507

Text #7

Max Range: 1.2328. Min Range: -1.2328

Max Act: 1.0543. Min Act: -0.0001

Data Index: 111186 (C4 (Web Text))

Max Activating Token Index: 682

Text #8

Max Range: 1.2328. Min Range: -1.2328

Max Act: 1.0534. Min Act: -0.0001

Data Index: 937956 (C4 (Web Text))

Max Activating Token Index: 797

Text #9

Max Range: 1.2328. Min Range: -1.2328

Max Act: 1.0460. Min Act: -0.0001

Data Index: 205344 (C4 (Web Text))

Max Activating Token Index: 301

Text #10

Max Range: 1.2328. Min Range: -1.2328

Max Act: 1.0627. Min Act: -0.0001

Data Index: 218620 (C4 (Web Text))

Max Activating Token Index: 686

Text #11

Max Range: 1.2328. Min Range: -1.2328

Max Act: 1.0555. Min Act: -0.0001

Data Index: 1225362 (C4 (Web Text))

Max Activating Token Index: 1017

Text #12

Max Range: 1.2328. Min Range: -1.2328

Max Act: 1.0377. Min Act: -0.0001

Data Index: 748053 (C4 (Web Text))

Max Activating Token Index: 687

Text #13

Max Range: 1.2328. Min Range: -1.2328

Max Act: 1.0428. Min Act: -0.0001

Data Index: 171611 (C4 (Web Text))

Max Activating Token Index: 514

Text #14

Max Range: 1.2328. Min Range: -1.2328

Max Act: 0.9598. Min Act: -0.0001

Data Index: 221731 (C4 (Web Text))

Max Activating Token Index: 113

Text #15

Max Range: 1.2328. Min Range: -1.2328

Max Act: 0.9464. Min Act: -0.0001

Data Index: 1314910 (C4 (Web Text))

Max Activating Token Index: 280

Text #16

Max Range: 1.2328. Min Range: -1.2328

Max Act: 0.9664. Min Act: -0.0001

Data Index: 1185627 (C4 (Web Text))

Max Activating Token Index: 101

Text #17

Max Range: 1.2328. Min Range: -1.2328

Max Act: 0.9717. Min Act: -0.0001

Data Index: 547208 (C4 (Web Text))

Max Activating Token Index: 909

Text #18

Max Range: 1.2328. Min Range: -1.2328

Max Act: 0.9295. Min Act: -0.0001

Data Index: 1039164 (C4 (Web Text))

Max Activating Token Index: 580

Text #19

Max Range: 1.2328. Min Range: -1.2328

Max Act: 0.9468. Min Act: -0.0001

Data Index: 562772 (C4 (Web Text))

Max Activating Token Index: 672