Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 602 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 2.8234. Min Range: -2.8234

Max Act: 2.8234. Min Act: -0.0000

Data Index: 263926 (C4 (Web Text))

Max Activating Token Index: 105

Text #1

Max Range: 2.8234. Min Range: -2.8234

Max Act: 2.6739. Min Act: -0.0000

Data Index: 1544607 (Python Code)

Max Activating Token Index: 875

Text #2

Max Range: 2.8234. Min Range: -2.8234

Max Act: 2.4565. Min Act: -0.0000

Data Index: 1437227 (Python Code)

Max Activating Token Index: 900

Text #3

Max Range: 2.8234. Min Range: -2.8234

Max Act: 2.6965. Min Act: -0.0000

Data Index: 1468045 (Python Code)

Max Activating Token Index: 103

Text #4

Max Range: 2.8234. Min Range: -2.8234

Max Act: 2.5034. Min Act: -0.0000

Data Index: 1444403 (Python Code)

Max Activating Token Index: 661

Text #5

Max Range: 2.8234. Min Range: -2.8234

Max Act: 2.4996. Min Act: -0.0000

Data Index: 1434892 (Python Code)

Max Activating Token Index: 815

Text #6

Max Range: 2.8234. Min Range: -2.8234

Max Act: 2.4970. Min Act: -0.0000

Data Index: 1442342 (Python Code)

Max Activating Token Index: 98

Text #7

Max Range: 2.8234. Min Range: -2.8234

Max Act: 2.4950. Min Act: -0.0000

Data Index: 1610883 (Python Code)

Max Activating Token Index: 222

Text #8

Max Range: 2.8234. Min Range: -2.8234

Max Act: 2.4824. Min Act: -0.0000

Data Index: 1389740 (Python Code)

Max Activating Token Index: 739

Text #9

Max Range: 2.8234. Min Range: -2.8234

Max Act: 2.7443. Min Act: -0.0000

Data Index: 1652219 (Python Code)

Max Activating Token Index: 294

Text #10

Max Range: 2.8234. Min Range: -2.8234

Max Act: 2.5619. Min Act: -0.0000

Data Index: 1363957 (Python Code)

Max Activating Token Index: 497

Text #11

Max Range: 2.8234. Min Range: -2.8234

Max Act: 2.6438. Min Act: -0.0000

Data Index: 1647217 (Python Code)

Max Activating Token Index: 687

Text #12

Max Range: 2.8234. Min Range: -2.8234

Max Act: 2.3963. Min Act: -0.0000

Data Index: 1625218 (Python Code)

Max Activating Token Index: 936

Text #13

Max Range: 2.8234. Min Range: -2.8234

Max Act: 2.4347. Min Act: -0.0000

Data Index: 1376131 (Python Code)

Max Activating Token Index: 210

Text #14

Max Range: 2.8234. Min Range: -2.8234

Max Act: 2.4273. Min Act: -0.0000

Data Index: 1633059 (Python Code)

Max Activating Token Index: 353

Text #15

Max Range: 2.8234. Min Range: -2.8234

Max Act: 2.2727. Min Act: -0.0000

Data Index: 1444671 (Python Code)

Max Activating Token Index: 590

Text #16

Max Range: 2.8234. Min Range: -2.8234

Max Act: 2.5397. Min Act: -0.0000

Data Index: 1560166 (Python Code)

Max Activating Token Index: 738

Text #17

Max Range: 2.8234. Min Range: -2.8234

Max Act: 2.2647. Min Act: -0.0000

Data Index: 1650298 (Python Code)

Max Activating Token Index: 357

Text #18

Max Range: 2.8234. Min Range: -2.8234

Max Act: 2.3631. Min Act: -0.0000

Data Index: 1550896 (Python Code)

Max Activating Token Index: 718

Text #19

Max Range: 2.8234. Min Range: -2.8234

Max Act: 2.2473. Min Act: -0.0000

Data Index: 1533471 (Python Code)

Max Activating Token Index: 670