Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 778 in Layer 11

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 2.3946. Min Range: -2.3946

Max Act: 2.3946. Min Act: -0.0000

Data Index: 1400795 (Python Code)

Max Activating Token Index: 951

Text #1

Max Range: 2.3946. Min Range: -2.3946

Max Act: 2.0928. Min Act: -0.0000

Data Index: 1383792 (Python Code)

Max Activating Token Index: 602

Text #2

Max Range: 2.3946. Min Range: -2.3946

Max Act: 1.6786. Min Act: -0.0000

Data Index: 1603253 (Python Code)

Max Activating Token Index: 512

Text #3

Max Range: 2.3946. Min Range: -2.3946

Max Act: 1.6619. Min Act: -0.0000

Data Index: 1546542 (Python Code)

Max Activating Token Index: 420

Text #4

Max Range: 2.3946. Min Range: -2.3946

Max Act: 1.4680. Min Act: -0.0000

Data Index: 1600484 (Python Code)

Max Activating Token Index: 246

Text #5

Max Range: 2.3946. Min Range: -2.3946

Max Act: 1.5072. Min Act: -0.0000

Data Index: 1400373 (Python Code)

Max Activating Token Index: 902

Text #6

Max Range: 2.3946. Min Range: -2.3946

Max Act: 1.4497. Min Act: -0.0000

Data Index: 1592883 (Python Code)

Max Activating Token Index: 556

Text #7

Max Range: 2.3946. Min Range: -2.3946

Max Act: 1.4897. Min Act: -0.0000

Data Index: 1576493 (Python Code)

Max Activating Token Index: 182

Text #8

Max Range: 2.3946. Min Range: -2.3946

Max Act: 1.4082. Min Act: -0.0000

Data Index: 1489406 (Python Code)

Max Activating Token Index: 546

Text #9

Max Range: 2.3946. Min Range: -2.3946

Max Act: 1.4621. Min Act: -0.0000

Data Index: 1502213 (Python Code)

Max Activating Token Index: 300

Text #10

Max Range: 2.3946. Min Range: -2.3946

Max Act: 1.1912. Min Act: -0.0000

Data Index: 1422065 (Python Code)

Max Activating Token Index: 484

Text #11

Max Range: 2.3946. Min Range: -2.3946

Max Act: 1.2566. Min Act: -0.0000

Data Index: 1626876 (Python Code)

Max Activating Token Index: 693

Text #12

Max Range: 2.3946. Min Range: -2.3946

Max Act: 1.2347. Min Act: -0.0000

Data Index: 1425538 (Python Code)

Max Activating Token Index: 556

Text #13

Max Range: 2.3946. Min Range: -2.3946

Max Act: 1.1770. Min Act: -0.0000

Data Index: 1516781 (Python Code)

Max Activating Token Index: 395

Text #14

Max Range: 2.3946. Min Range: -2.3946

Max Act: 1.1571. Min Act: -0.0000

Data Index: 1543297 (Python Code)

Max Activating Token Index: 221

Text #15

Max Range: 2.3946. Min Range: -2.3946

Max Act: 1.2200. Min Act: -0.0000

Data Index: 1471176 (Python Code)

Max Activating Token Index: 442

Text #16

Max Range: 2.3946. Min Range: -2.3946

Max Act: 1.1616. Min Act: -0.0000

Data Index: 1547410 (Python Code)

Max Activating Token Index: 313

Text #17

Max Range: 2.3946. Min Range: -2.3946

Max Act: 1.2064. Min Act: -0.0000

Data Index: 1599274 (Python Code)

Max Activating Token Index: 995

Text #18

Max Range: 2.3946. Min Range: -2.3946

Max Act: 1.0570. Min Act: -0.0000

Data Index: 198719 (C4 (Web Text))

Max Activating Token Index: 592

Text #19

Max Range: 2.3946. Min Range: -2.3946

Max Act: 1.1176. Min Act: -0.0000

Data Index: 1514438 (Python Code)

Max Activating Token Index: 706